34 - iassisi quarterly

Access To
Unpublished
Social Science

Data in

Government
Departments

in the

United Kingdom

by D. A. Clarke '

Both for the purposes of day-to-day

administration and for the formulation of future

policy, government departments gather data on a

wide range of topics which are potentially

valuable for social science research. (Social

science is here interpreted very broadly, to

include economics and government as well as

social studies in the more restricted sense.) A
few random examples are agricullural economics

and farm profits; consumption of food and

drink; public service pay and personnel

management; social planning; regional and town

planning; population trends; the supply and

training of teachers; employment statistics;

labour mobility; industrial relations; immigration

and race relations; safety, health and welfare;

manpower planning; local prisons and penal

policy; the police. The records of some other

quasi-official bodies, e.g. the research councils

and the nationalized industries, must similarly

contain data of major value for research.

'Presented at lASSIST/IFIX) International
Conference May 1985, Amsterdam

Though some types of material clearly need to

remain closed for a considerable period (e.g.

census material and other records of individuals,

and data supplied in confidence by business)

most of it is not inherently confidential and

could be made accessible to bona fide

researchers long before the expiry of the 30

years after which government records normally

become accessible in the Public Record Office

(PRO). The problem for the researcher is first

to discover what data exists and then to obtain

access to iu

In the 1940's and early 1950's the

Inter-Departmenial Committee on Social and

Economic Research (the North Committee) put

in hand extensive listings of government

material of social science interest (including

some unpublished material) held by various

departments, which resulted in the publication

of a valuable series of six "Guides to Official

Sources". The Social Science Research Council,

soon after it came into being in 1965, set up a

Social Science and Government Committee,

which served in a sense as a successor to the

North Committee. This latter committee

speedily recommended that the North

Committee initiative be repealed and the guides

brought up to date. The Council therefore

awarded a contract to the British Library of

Political and Economic Science (at the LxDndon

School of Economics) to produce a new, single

guide to all the unpublished government data

likely to be of interest to research workers in

the social sciences, and potentially accessible to

them, that a small team could locale and list in

a limited period. The resulting "Guide to

Government Data", published in 1974, reported

the material revealed by surveys of 17

departments, carried out over a period of three

years from September 1969. It is clearly of the

greatest importance that these listings should

now be updated, and the remaining departments

covered.

Although sheer lack of lime necessitated the

exclusion of some departments, the absence on

Fall/Winter 1985



iassist quarterly 35

other grounds of certain key departments is

greatly to be regretted; and it is important that

their records should now be surveyed. In

particular, the Department of Trade and

Industry did not find it possible to cooperate in

the enterprise; and, though a great deal of

material is published by the department, a guide

10 unpublished material there remains a most

important desideratum for economists. The

Foreign and Commonwealth Office was

excluded in view of the confidentiality of much
of its unpublished material and the practical

difficulty of sorting it; those researching

contemporary history will nevertheless find it

particularly unfortunate that such materials as

press releases could not readily be assembled

and made available to serious scholars. The
records of the former Department of Scientific

and Industrial Research contain a vast fund of

valuable data relevant to the social sciences in

many fields; but preliminary approaches made it

clear that a worthwhile survey was impracticable

at that time.

Though problems of personal privacy and

commercial confidentiality will probably always

prevent access to recent unpublished data in the

Office of Population Surveys and H.M.

Customs and Excise, the remaining departments

should now be included in a new survey. It

seems very possible that in some cases the

decision that a department could not cooperate

in the earlier project may have been afTecied by

traditional department, or even by personal,

attitudes towards public access to official

documents per se\ or by the fact that its library

was particulariy hard pressed at the lime of the

enquiry: so that a renewed approach to the

department might now be favorably received

and a proper survey carries out It is also to be

hoped that the official attitude towards public

access is becoming more liberal, and that the

policy expressed at the time of the former

enquiry by the Civil Service Department,

namely that it should "make available as much
of the material as it reasonably can within the

official limits most generously interpreted" is

more widel>' accepted, so that it will be possible

to fund the compilation of a third Guide which

would not only update the surveys already

published but also reveal important sources of

unpublished data held by departments not

previously surveyed.

For access to material restricted by the 30-year

rule the researcher is normally dependent on
the libraries of departments. The amount, type

and quality of unpublished material actually

made available to a researcher in a library

varies considerably from department to

department, according to the extent to which it

is departmental policy to transfer papers from

administrative divisions or branches to its central

library. In turn the libraries have varying

policies on the admission of researchers, and the

extent to which readers, once admitted, are

given access to papers, as well as on publication

- some departments publish a substantial

proportion of the available material.

It may be useful to explain that working papers

in a government department are usually held in

individual registries attached to the separate

divisions or branches which they serve.

Although files are reviewed by the department

after five years in order to arrange for the

destruction of ephemeral material, files

containing correspondence and minutes are

retained by the department and are not

accessible to the public as of right imtil they

are transferred to the PRO after 30 years; and

some may be retained by the department after

that date on administrative groimds. So if

certain information, although not "classified" in

the official sense of security classification, is

neither readily available in a departmental

librar\ open to the public, nor published, nor

(as is usual in the case of statistics) provided on

payment of a fee to the statistical department

concerned, then it generally seems unlikely that

under present arrangements members of the

public will be able to consult the papers until

they have been transfened to the PRO.

Fall/Winter 1985



36 - iassist quarterly

It is clear thai a substantial proportion of the

statistical and other data gathered by

government departments is being, or could be,

recorded and analysed by automated data

processing methods. Both the actual progress

made in this area and the scope for further

development will no doubt be discussed in the

course of this conference. Meanwhile, in 1981

the Committee on Modem Public Records (the

Wilson Committee) recommended (para. 423)

that "the nucleus of a data archiving centre"

should be established as soon as possible; and

the Government agreed (Cmnd 8531 para 55)

that a feasibility study should be carried out for

the establishment of such a centre at the PRO.
It is to be hoped that any such centre will work

in close cooperation with the Social Science

Data Bank of the Economic & Social Research

Council.

The advent of data prcicessing techniques should

also enable scholars to exploit records more
fully and make new connections through the

adoption of machine indexing. This should also

assist with the hitherto intractable problem of

"particular instance papers" (PIPs). These

consist of groups of papers, often very

numerous indeed, the subject of which is the

same in all cases, though each individual paper

relates to a different person, body or place.

While each individual paper may in isolation be

of little importance, when a set of PIPs is taken

as a whole and analysed it may form a source

from which conclusions as to historical,

economic or social trends may be drawn.

In 1979 the appointment of the Wilson

Committee provided an opportunity to propose

changes which would facilitate the use of

government archives by researchers in the social

sciences. Some such opportunities which were

taken, and others which were missed, are set

out below.

In view of the potential importance of data

contained in the records of such bodies as the

research councils and the nationalized industries.

it was recommended that the PRO should make
proposals for consistent arrangements for the

selection, preservation and public inspection of

the records of the research councils and

nationalized industries.

Substantial concern had been expressed to the

Committee about the "weeding" of records

before they are transfened to the PRO. It was

felt that coherent, clearly applicable objectives

should be established; and that those carrying

out this task needed to have an appreciation of

the needs of present and future scholars. It was

also suggested that some types of "weeded"

material might be transferred for preservation

outside government departments (e.g. to

universities); and that special note should be

taken of the importance for research of the

working, papers of Royal Commissions and

Select Committees and the evidence presented

to them. The Committee recommended the

adoption of a system of "sector panels",

including both departmental interests and

researchers with experience of the field in

question, to advise departments on the official

and wider purposes to which departmental

records might be put, so that this advice could

be taken into account by the "weeders"; but the

Government has rejected this recommendation.

The Committee made no reference to the

suggestion that some "weeded" material might

be preserved outside the relevant department: it

was perhaps thought that the organization and

exploitation of such deposits might require

rather substantial expenditures by the recipient

organizations.

The Committee made no reference to the

suggestion that the press releases issued by

Government departments should be

computerized; nor to the concern that has been

expressed over the rigid application of the

"30-year rule" when material on the same topic

is already accessible outside the PRO and when
material of a routine nature, but of some

historical interest, is automatically restricted

under the rule.n

fall/Winter 1985



iassist quarterly 37

New Products

Data User Services Division

Data Developments
BUREAU OF THE CENSUS
U^. Department of Commerce
Wathington, D.C. 20233

Census Ofl^ers International Data Base on Diskette

The U.S. Bureau of the Census is now offering its International Data Base (IDB) on disicette for

203 countries of the world. Organized as a series of 94 statistical tables, the IDB contains

demographic, economic, and social data for all countries of the world. Each table is fully annotated

with information on sources, methods of collection, definitions of terms, methods used in

computations, and qualifications of the data.

Each diskette contains tables and notes for a single country. A computer program, written in Basic,

is supplied to users to extract individual tables into formats acceptable to software packages such as

LOTUS 1-2-3, dDBASE, SUPERCALC, and SPSS[sic]. File documentation describing the contents

of the tables and notes is also included. The cost for the first country purchased on diskette,

including file documentation, is $15. The price for each additional country is $7.50.

The statistical tables from the IDB are also available in printed copy. When ordering printed copies

of statistical tables, the first 10 tables are free and additional tables are $0.25 each.

On-line access to the IDB is available to users within the federal govemmenL Currently, there are

56 organizations within the federal government that have on-line access to the IDB through computer

terminals or microcomputers with dial-up capability.

The data contained in the IDB are collected from many sources, including national statistical offices

throughout the world, international organizations such as the United Nations, research centres and

universities around the world, and federal agencies such as the U.S. Agency for International

Developmenu Data that originate in host countries usually come from census, vital registration

systems, surveys, and administrative records.

Fall/Winter 1985