vol29-4.indd IASSIST Quarterly Winter 2005 by by Julie Lamb* Disseminating Survey Information in the Networked World:A UK Resource Abstract Survey researchers are increasingly turning to the WWW in an effort to find information for the data collection stage of their projects as well as for the more traditional activity of searching for literature and reports. This paper will discuss the development and use of the Question Bank (Qb), an innovative WWW resource which is used to teach students and researchers about UK social surveys produced by survey agencies such as the Office for National Statistics and the National Centre for Social Research. The Question Bank contains the full questionnaires for over 50 social surveys and is continually expanding. These questionnaires enable researchers to take questions that have been used in large scale surveys for use in their own research work, thus ensuring that they do not spend time ‘re-inventing the wheel’. The Qb also contains information on social measurement in 22 substantive topic areas, and has numerous resources relating to survey data collection methods. The resource is free to all. Introduction This paper is about the Question Bank (Qb), a free UK web based resource which helps to disseminate questionnaire metadata to researchers and students’ wishing to see what has been done before. Contrary to what the title of the 2006 IASSIST conference; ‘Data in a World of Networked Knowledge’ suggests, there is not a huge amount of easily accessible data concerning the network or the knowledge of social science survey researchers, especially those involved with professional survey research. Even fewer data exist on how such students and researchers use the knowledge that they acquire through the Internet. One survey which has studied exactly how students at college use the Internet has concluded that: “Internet use is a staple of college students’ educational experience. They use the Internet to communicate with professors and classmates, to do research, and to access library materials. For most college students the Internet is a functional tool, one that has greatly changed the way they interact with others and with information as they go about their studies.” (Jones et al 2002:2) The question is then, how do we as social scientists or information professionals provide students with relevant and useful information in a user friendly web interface? The Question Bank has attempted to do this since 1996, a time when there were very few online resources available, especially in the survey research world. One notable exception is the UK Data Archive which has been disseminating data since the 1960s, and has had a web presence since 1995. The Qb and the UK Data Archive now work closely together to ensure that the future for UK survey researchers and secondary analysts is one in which a seamless interaction between survey web services is possible. In a previous edition of IASSIST Quarterly, Margaret Law used the phrase ‘Reduce, Reuse, Recycle’ (Law 2005:5) and discussed issues surrounding the phrase when dealing with data. The Qb also recycles, but does so with metadata, mainly questionnaires from major UK probability surveys. The idea behind the Qb is that those wishing to write their own survey questions do not have to begin from scratch when there are many questions already in use and the accompanying coding frames and documentation can be easily accessed The Question Bank: A Brief History As stated above, the Question Bank was developed starting in 1995 with a grant from the UK Economic and Social Research Council. The Question Bank was established as part of the UK Centre for Applied Social Surveys1, and sought to contribute to strengthening the quality of UK survey research and in particular to try to improve survey measurement. In addition to providing access to questionnaires, many of which were not readily accessible previously, the Question Bank attempts to present commentary, written by experts, on the quantitative measurement of different survey variables, as well as to provide access to the Harmonisation project of the Office for National Statistics which seeks to standardise survey measurement across government surveys on a number of variables.2 The approach adopted by the Question Bank has been 6 IASSIST Quarterly Winter 2005 an inductive one, that is, to make a very large amount of material available for easy access on the WWW, and to invite the user to search this material using a powerful search engine on the Qb site. This requires ingenuity and insight on the user’s part in order to bridge the gap between concept and variable, or between the abstract idea and the actual question. In some cases this is easier to achieve than in others. The Topic commentary is intended to sensitise the user to some of the issues which can arise, but the amount of material on converting concepts into variables is not extensive. The Question Bank relies upon the user drawing their own conclusions about the comparability of questions and the most appropriate questions to include in the type of survey which the user is designing. One of the key problems in UK survey research is that the large scale surveys mainly funded by the Government (which are used for official statistics) are conducted by survey agencies outside academia. These agencies have large staff numbers and budgets. This means that academics, especially students, are far removed from the real world of survey research. The Qb has been developed with the aim of helping three main groups. Firstly, researchers devising their own survey questionnaires, by providing easily-accessed illustrations of how the topics with which they are grappling have been handled/measured in professionally designed surveys. Another group which the Qb aims to help is secondary analysts of survey data, either at the stage at which they are seeking out surveys containing material of interest to them, or at the stage when, having worked with particular survey data sets, they wish to learn more about the underlying survey processes and their likely strengths, weaknesses and limitations. Finally, we aim to help teachers and students of survey methods, by providing text and examples on the wording of questions and the construction of questionnaires and examples from large surveys already carried out. The Question Bank has now grown into a very large resource with over 40,000 pages of PDF questionnaire material in around 4000 files! In 2005 funding of the resource was reorganized by the Economic and Social Research Council, and it became a separate entity from the CASS model, with greater funding for more staff. This will ensure that the Qb continues to grow and that essential changes to the look and feel of the web site can be managed. The Web Site The Qb is a very large site organised in a flat file structure. Materials are organised around three main themes/areas of the site: Surveys, Topics, and Resources. I will now go through each of these in turn. Surveys The Surveys area is the largest part of the Qb and houses not only the questionnaires from large scale UK social surveys but also supplementary information about those surveys. This ensures that Qb users can see what a real survey questionnaire looks like and can find out how a survey is carried out from start to finish. For example, for each survey the Qb has extensive links back to the organization that carried out the survey and to technical and final survey reports that give details about the timelines for the survey, the methods used, sampling techniques, advance letters sent out, coding frames, how the data were analyzed , as well as the final results. It is therefore possible, with the questionnaires and information on the Qb and the links to other resources such as the Department of Health and the Office for National Statistics, for students to run mini replica projects to really find out how survey research is carried out, and then use other resources such as NESSTAR3 at the Economic and Social Data Service4 to attempt to analyse the actual data from the survey at no cost. Currently the Qb holds information on 57 UK social survey series and is continuing to expand. This amounts to over 40,000 pages of questionnaire material, all of which is fully searchable using our advanced search engine. The surveys for which questionnaires are included in the Question Bank are all social surveys using a probability sample design. Commercial market research surveys and business surveys directed to organisations are not generally included in the Question Bank. In most cases the population units that the surveys we focus on are intended to study are either individual persons, or domestic groups such as households or families. These surveys deal with a very wide range of topics which relate to the circumstances, behaviour and attitudes of these units. The Qb focuses mainly on large-scale quantitative surveys. Most of them have quite long and complex questionnaires that are administered in the field (or by telephone) by trained social survey interviewers, the type of survey that would be too vast for a student to replicate for their own projects. A high proportion of the in-scope surveys have been conducted either by, or for, central government departments. Others are major academic surveys. Many are repeated continuous or longitudinal surveys which have produced annual series of published results. Another main reason for selecting the questionnaires of particular surveys for inclusion in the Qb are that these surveys on a national scale are generally treated as benchmarks against which other surveys in the same topic areas can be compared. The criteria for selecting surveys as benchmarks are: • that the survey should have been professionally de- veloped and conducted to a high technical standard; • that it should cover a national reference population; IASSIST Quarterly Winter 2005 7 • that it should be a prime current source of informa- tion on important social science topics that the Qb sets out to cover. Given their origin, it can be assumed that the questions reproduced in the Qb have also been pilot tested. Therefore they are likely, on the whole, to perform better as a means of collecting quantitative information for particular purposes than questions which someone coming fresh to a survey topic, without previous question drafting experience, might devise for themselves. The Question Bank aims to keep up with the constantly increasing tempo of new questionnaire instruments coming on stream, and the release of survey datasets to the UK Data Archive. Retrospectively, we decided to try to cover the period from 1991 (a Census year) onwards, but not to attempt systematic coverage of the period before 1991. However, a number of surveys conducted before 1991 are still used as benchmarks, or exemplify particular innovations in question or data collection design. For these we have made exceptions to our rule, so that the Qb contains questionnaires for selected surveys conducted during the 1980s. Maintaining this material is not an easy task. One full time Content Manager works to identify, update and add new material to the site. Each questionnaire that we obtain has to be fully marked up in Adobe Acrobat, bookmarked, and linked to all related material, all of which takes time. However, this task is essential if material is to be easily accessed by Qb users. Topics The topics section of the Qb site is aimed at assisting researchers with social measurement. As Oppenhiem states; “The questionnaire has a job to do: its function is measurement” (1992:100). The Qb clearly has questionnaires from large scale surveys that researchers can use to measure phenomena. However, the topics area of the Qb goes further than this and aims to take 22 key social science topics and provide commentary on how social measurement can be done within them. These 22 topics are: · Crime and victimisation · Demography · Economic activity · Education · Ethnicity and race · Family · Gender · Geography · Health, illness and disability · Housing and household amenities · Household definition and structure · Income, expenditure and wealth · Leisure and lifestyles · Political behaviour and attitudes · Religiosity · Social attitudes in general · Social capital · Social class · Social protection and care · Travel and transport · Voluntary associations · Working life Each topic has a table detailing surveys that contain questions seeking to measure that phenomenon. For each topic area, we aim to provide a summary account of the main concepts involved and current approaches to measuring those concepts using quantitative methods. Further, an aim from the beginning of the Qb has been to have specially commissioned commentary written by experts in the field on measurement of that topic. The commentary is intended to help users to understand the conceptual structure of each topic area and the way it is reflected in the structuring of questions. It may also make users aware of other concepts and questioning approaches that may be closely related to the ones that they had in mind when accessing the Qb. The commentary includes discussion of available objective evidence on the validity and reliability of the measures produced by questions used. This has been problematic for a number of reasons; the main one is that such experts do not have time spare to write. Another problem is the nature of the Qb itself. As a free online resource, academics do not get recognition from their peers for writing for the web. Having said this, the Topics area is very well used and is especially popular with those that are beginning measurement of a new topic and who are looking for further guidance. We are making every effort to ensure that this area is as well populated as possible. An editorial board has been set up to monitor the quality of material in the Qb, and to commission commentary from experts in their respective subjects. This will further the aim of building up within the Qb site an electronic encyclopaedia of the social survey world. In addition we provide bibliographic references to relevant social science research literature on each topic, and hyperlinks to other Internet sites containing relevant information. 8 IASSIST Quarterly Winter 2005 Resources This section of the site houses a number of different resources which researchers may find useful for their survey research. As well as information about the Qb, including teaching materials, user guides and historical documents, this area includes an explanation of the following resources: Harmonised question forms The Office for National Statistics (ONS) initiated a programme of work and negotiation that aimed at arriving at a set of variables with question wording harmonised across surveys. The result was a booklet entitled ‘Harmonised Questions for Government Social Surveys’. These harmonised wordings have now been, or are in process of being, adopted in all major continuous government social surveys and are feeding into work on the new Integrated Household Survey. CAPI Documentation IASSIST members will be well aware that most contemporary large-scale interview surveys are carried out by the interviewer carrying a portable computer on which the questionnaire resides as a program for Computer Assisted Personal Interviewing (CAPI). This represents a major technical advance in the survey process, but also poses a challenge in making the CAPI interview intelligible to the layperson. Students in particular are often unaware of CAPI and the implications it has for questionnaire design and administration. The Qb team is working to make the way CAPI surveys work in the field as transparent as possible to Qb users. Survey Link Scheme The ESRC Survey Link Scheme exists to give academic social scientists the opportunity to acquaint themselves with professional social survey research, carried out by the Office for National Statistics, the National Centre for Social Research and various market research companies. It thus provides a bridge between academia and the practical world in which professional survey research is carried out. Two linked components are offered, each typically taking one day each: ·• One-day workshops, which provide a briefing on a particular survey. The day includes an introduction to survey interviewing in the field, an introduction to CAPI – Computer Assisted Personal Interviewing — and guidance through the CAPI questionnaire of a particular survey by professional staff from the agency carrying out the fieldwork. These workshops are held at various locations throughout the UK ·• The opportunity to go out with a professional inter- viewer for one day, and observe a social survey inter- view in the field. This can be arranged close to where the scheme participant lives, and takes place after atten- dance at the workshop. The website for the scheme is maintained by the Question Bank and Professor Martin Bulmer directs both resources. Further resources are planned for the Qb in the coming few years. These include new fact sheets about the survey process, and further information on how researchers can utilise new technologies in their own small scale research. Searching the Qb In March 2006 the Qb implemented a new advanced function for our APR Smartlogik search engine. The search engine was specifically chosen because it searches within PDF documents as well as html pages. The new advanced search makes it possible to search in the following ways (taken from the Qb search help pages): · Search with ALL the words Use this section if there are several different words that are relevant and you want to find only documents that include every one of these words but not necessarily in any particular order. This reduces the number of results and should narrow the search to more relevant ‘hits’. Example: A search for “pollution” will generate over 120 hits while a search for “pollution, health” in this section generates around 35 hits (March 2006), these documents will have elements involving each word although not necessarily in the same sentence or question. · Search with ANY of the words Use this section if you have several different words that may be equivalent to each other, maybe as alternative descriptions of the same concept, and you want to find documents that include only one, two, or more, of them. Example: Search in this section for “marijuana, heroin, cocaine” to find questionnaires about illegal drugs ( a search for the word “drugs” would also find prescription drugs mentioned in health surveys). · Search with the EXACT PHRASE This search is similar to “Search with ALL the words” above but with the addition that the order of the words in the search enquiry becomes significant. Please note that the search engine may not use every single word of the exact phrase to find the hits, it ignores the little words like “of”, “and”, “are” or “with”, and so these may not be highlighted in the results. However, provided the phrase you have searched for is reasonably complicated you should have mostly correct hits. Example: Search in this section for “people like me have no say in what government does” to find all the surveys that have used this particular question about political efficacy. · Search WITHOUT the words This section should be used together with an ALL words or IASSIST Quarterly Winter 2005 9 an ANY words search. Sometimes a term you want to find has more than one use or meaning, so you would use this section to try to exclude the hits that involve the meanings that you do not want to find. Do this by putting in this section some words that would often occur close to the search word when it is being used with the meaning you do not want to apply. Example: The word “train” could refer to a form of transport or an education process. To find materials about the form of transport enter “train” in the ALL words search and “learn, program, vocation, job, college” in the WITHOUT the words section. You will need to add more terms to the without section to eliminate every unwanted hit but this should convey the idea. · Combining sections in the search command It is not possible to combine an EXACT PHRASE search with a WITHOUT the words search. But you may combine an EXACT PHRASE search with either an ANY words or an ALL words search. Example: Repeat the “marijuana, heroin, cocaine” ANY words search from above but combine it with an EXACT WORDS search for “have you ever tried” and then again with “have you ever taken” and compare the two sets of results. · Search by Survey Another way of eliminating unwanted results is to use the survey titles to select just the surveys that you would expect to be relevant to your search. You may select more than one survey to be included by holding the Control key down as you make selections from the scrolling list using your left mouse button (Mac users should hold their Apple key down while making selections with their mouse button). Do not forget to clear these selections by clicking on the ALL item at the top of the scrolling list before starting a new search. · Search by Questionnaire Year It may be possible to restrict your search further by limiting it to a specific year. However this particular element may also prove to be unreliable in some circumstances due to a lack of consistency in recording survey year numbers. Where a survey is identified with single calendar years (eg. 1997) this function should work properly. But where a survey is recorded as “97/98” or “97/8” there is less likelihood of this function working well. In some cases selecting 1997 will generate hits in surveys titled 1997/98 and 1996/97. · Search by Document Type When you have become familiar with the Question Bank you may find this option more useful to limit the types of material offered to you in search results. All of the original materials prepared by the researchers and survey organisations that can be found on this website are stored and displayed in PDF files (which can only be opened by using Adobe Acrobat Reader). These are mainly questionnaires but also include show cards, advance letters, interviewer instructions and technical report extracts. Most of the materials prepared by the Question Bank staff to help users locate particular surveys or to provide background information on survey methodologies, topic guides, links and bibliographies, are stored and displayed as HTML files (web pages). Interpreting the Results of a search Each hit result is headed by a document title which should include a good indication of the type of document that has been found and, if it is a questionnaire extract, the particular survey series and year applicable. If your search terms have been found in several different parts of a document you will probably be shown a separate hit for each page involved. On the results list you should be able to see the successful search word highlighted in red with about 50 words to show the context in which it has been found. This should help you to select the hits which are most likely to be useful to you. When you click on a particular hit to follow the link there will be a slight delay while the file is downloaded to your computer, after which you will see the beginning of the document on your screen. There may then be another pause before the display changes again to show the relevant section of the file with the search words highlighted in blue or grey. You may want to scroll up or down the document to read more material in the vicinity of the located result, and you may find it helpful to zoom in or out to see the text more clearly or to see its context on the page. To return to the search results, click on the “Back” button on your browser toolbar. The document title that you have just opened should now be displayed in a different colour to that of documents you have not followed-up. Maintaining the Resource As I have already stated, the Qb is a large site which is maintenance intensive. Until November 2005 when further funding was received for a team of four staff, the Qb was run from the University of Surrey, Department of Sociology with a complement of 2 staff, one of whom was part time. This insufficient funding is a common problem for research projects. However I mention it here because it has become clear, now we have a larger team, just how much we could do with the Qb in an ideal world. One of the most problematic areas has been the technical knowledge that we have at times desperately needed. From the beginning the Qb has been a social science project rather than an IT project. All of its team members, with the exception of one past manager, have been social researchers. It is a constant struggle to keep up with the 10 IASSIST Quarterly Winter 2005 latest web developments, with developments in server technology and to fix technical issues. In short, the web development side of the Qb has been a very steep learning curve for the team, which we will continue to struggle with whilst attempting to keep the balance between a comprehensive resource on social measurement and an easy to use web resource. The main priorities that we have for the next three years are to keep updating the site, particularly the questionnaires and the Topics; to conduct a thorough literature review of CAPI and possibly conduct further research on this; and to enhance the resource through adding some limited European content. We are also hoping to be able to redesign the web pages with a more modern and user friendly look. The Qb is also gaining experience and expertise in questionnaire archiving processes, especially with the Data Documentation Initiative. Clearly the materials that the Qb houses are produced by external agencies and are largely untouched by us, except for the addition of bookmarks and explanatory material about the survey. Organisations with which we work closely, such as the UK Data Archive, have a great deal more technical expertise than we have in initiatives like the DDI and in techniques for digital preservation of data (and metadata), and it is envisaged that we will be working much more closely with the Archive in the future. We are therefore extremely pleased to be involved in this IASSIST conference and the following DDI meeting, and we welcome comments and discussion with IASSIST members. Conclusion Students and researchers are increasingly turning to the web in order to find materials for their work. The Qb is a large web resource aimed at social science researchers, students and teachers of research methods and secondary analysts. It contains information about large scale UK social surveys and their questionnaires, which enables users to recycle questions in their own work, thus avoiding having to reinvent the wheel. It also aims to have commentary on social measurement in 22 social science topic areas and various other resources linked to survey research. I have explained that the Qb is intensive to maintain, but that with additional funding from the ESRC, progress is being made towards making the Qb into a useful and user- friendly resource for the future. Useful Links: APR Smartlogik: http://www.aprsmartlogik.com ESRC Question Bank: http://Qb.soc.surrey.ac.uk ESRC Survey Link Scheme: http://Qb.soc.surrey. ac.uk/sls.htm Economic and Social http://www.esrc.ac.uk Research Council: Economic and Social Data http://www.esds.ac.uk Service: Office for National Statistics: http://www.statistics. gov.uk National Centre for Social http://www.natcen. ac.uk Research: NESSTAR: http://www.nesstar.org UK Data Archive: http://www.data-archive.ac.uk References: Jones, Steve (2002) ‘The Internet Goes to College: How Students are Living in the Future with Todays technology’ Washington, Pew Internet and American Life Project Available online: http://www.pewInternet.org Law, Margaret (2005) ‘Reduce, Reuse, Recycle: Issues in the Secondary Use of Research Data’, IASSIST Quarterly, 29, Volume 1 2005 pp 5-10 Available online: http://www.iassistdata.org/publications/ iq/iqvol29.html Oppenhiem A.N (1992) ‘Questionnaire design, Interviewing and Attitude Measurement’ London, Continuum Acknowledgements: I would like to thank Graham Hughes, Laura Hyman and Zoe Tenger for their comments. * Paper presented at the IASSIST 2006 conference in Ann Arbor, Mich., session: “Moving Beyond Data to Networked Knowledge”. Contact: Julie Lamb, Department of Sociology, University of Surrey, UK. Telephone: +44 (0)1483 683762, E-mail: Julie.lamb@surrey.ac.uk Footnotes 1 http://www.socstats.soton.ac.uk/cass - CASS ceased to exist in 2005 when the Question Bank and the Short Courses were refunded as separate entities. 2 http://www.statistics.gov.uk/about/data/harmonisation/ default.asp 3 NESSTAR is an online tool which enables basic data analysis. It is used by the Economic and Social Data Service to display key UK social survey data sets such as the General Household Survey. 4 The UK Economic and Social Data Service (ESDS) works to provide access to UK survey data and to support its use. There are five services within ESDS each with a remit for data support in the specialist area; ESDS Government, ESDS Longitudinal, ESDS International, ESDS Qualidata and ESDS Access and Preservation. All can be found online: http://www.esds.ac.uk