lASSIST Newslettert Vol. 3« ^o. t (Fall 1979) DATA ARCHIVES RETROSPECTIVE AMD PERSPECTIVE- N. K. Nijhawan It is sometimes useful* in order to gain an understanding of the prob- lems and opporunities for data archival efforts* to review the experi- ences of others. Especially the problems encountered in establishing archival activities In "Third World" nations may be illuminating for oth- ers* especially those used to the general level of technological deploy- ment in the United States and Europe. N . K. Nijhawan has provided a des- cription of current machine readable data archival efforts in Inaia. Editor. The out line Counci L for the ence da attempt work do makes s the dev during years, deta -e 1 ssues f data ar factors establi institu during will be obj e the of dev ta s t ne s ome elop the B d th chiv th shtne tlon the bri ct pro Sod elop arch exa fa tent ment cou ut disc e me es 1 at nt s a pas ef Ly of t gram al ment Ives mine r in ativ of r se bef uss i anin n ge ha V and II t tw dea his pap me of t Science of so In In criti this r e prop this of the re st on g and p neral a e led growth over t decad It with er he I Res cial di a* call egar osa L prog nex arti f urpo s we to of he es is to ndlan earc h sc 1 - It y the d and s for r amme t few ng a these se of U as the such world or so In the most general sense* a data archive is a Library of data. Like libraries of books* which con- centrate on the acquisition and cataloguing of books In order to make them accessible to the aca- demic community* these institu- tions* known variously as data archives* data banks* data resources' libraries* etc.* have been set up for the acquisition* or ga data need nity ever arch ma t 1 duri 1960 aval the n1ty gain deca USA r ap i comp and in s tate ot he data tist grow 1nst pres of t niza to s of . u * Ives on ng s w labl sod ed n des and d de ut er use ocia d by r . res s ha th itut umab his tlon meet the nlike the ism of d the hen e to al sc Thes oment or so Ues velop tech of q I sci thi Furt ource s al and 1 ons. progr and res soc usu ori g ore at a late the a s dis earc ial al I in rece arc 195 CO izab 1 ence um d « P tern ment nolo uant ence s te her* s am so c deve T Inf I amme Inst urln arti Eur s In 9/ o itat re chno the ong ont r lopm hese uenc in semi h an sci e ibra f t nt. hive Os mput le esea 1tut g th cuLa ope* th n th 1 ve sear Logy urg soc ibut ent dev ed Indl nat 1 d tr nee r 1 es hese Th s s and e r port rch 1 ons e pa rly d e fi e on tech ch f » e 1 laL ed of eLop sett on of a i n 1 ng commu- * how- data e f r- tart ed early became ion of commu- ha ve st two in the ue to eld of e hand ni ques add- on the share sden- to the these ment s * ing up Reprinted from ICSSR Newsletter Volume VIII (3 & 4)* October, 1977 - March, 1978* pp. 1-7* by permis- sion of N. K. Nijhawan. BACKGROUND The V.K.R.V. Rao Commltte which recommended the establishment of the Council also made a recommenda- tion that one of the functions of the Indian Council of Social Sci- ence Research should be "to develop and supoort centres for maintenance ana supply of data." In pursuance lASSIST Newsletter* Vol. 3, No. 't (FaU 1979) of th set u insti evoLv progr group 1970 datio which data the I few count nated These by Archl off 1c in 19 Is r P a tuti 1 ng amme sub and ns « con ceL CSSR se Le ry ne rec the ves e of 73. e c om wor ona L an e In mitt made th c e rn I" an ctea for twor omme Cou was the menda t 1 on » king g roup a rrange ffectlve d the cou ed Its rep a number e most 1 ed the set 1 n the se d the ass Inst 1tut1 developing k of dat ndat 1 ons w nc 1 1 and estabUs Counc 1 L the to ment s at a a nt ry . ort 1 of re mport ting creta Istin ons a a ar ere a the hed 1n Ne Counc 1 I suggest for r c h 1 va L The n March c ommen- ant of up of a riat of g of a In the coordi - ch 1 ves • pp roved Data In the w Delhi and get better returns from this Infrastructure. Briefly stated* the Council defined the functions of the Data Archives as follows: 1. to acquire* organize and maintain social science data sets 1n machine- readable form and make them available to inter- ested social scientists for re-use* 2. to support suitable research institutions in different parts of the country to develop simi- lar institution-based data archives* FUNCTIONS Th Sclen uncon t ions vent 1 so I. la posed semin for r tant be pe of th the must t Ions of t Insta train probl by ma Equal of e adequ soc la of th that only of da tion f unct e In ce R vent 1 of 1 onal I I sci to a ate m e-use f unct r f orm e ICS view assum 1 n V he I nce» ed m em of Jor ly 1m asy ate I sci ese p the perf o ta ac but ions di an esea onal ts y« ence cqui achi ions ed SR. tha e s lew ndi a we anpo ace dat a port acce soft ence rob I Data rm c qui s also to h Co r ch vi Data as m dat re* ne-r Thes and by t Th t th ome of t n hav wer . ess pro ant ss ware re ems* Arc onve itio ta elp unc 1 1 has ew of Arch i V ent lone a archi organi z eadab le e are V were p he Dat e Coun e Data addit io he perc situati e a sh Ther to dat due ing Is the f comp facil search, it w hives nt ional n and ke up bulla a of S taken the es. d ab ve is e and dat a ery 1 ropos a Arc cil w Arc nal ul i ar on. ortag e is a pro agen que ut ing ities Be as de shou I f unc di sse addit die oc 1 a I an f unc - Con- ove a sup- dis- set s mpor- ed to hives as of hives f unc - ities For e of the duced c ies . st ion and for cause c i ded d not 1 1 ons m 1 na- 1 ona I nt e I e 3. to develop and maintain a computer programme library to handle the data management and retrieval problems of the data archives as well as data processing and ana- lysis of problems of social scientists* 4. to organize training courses in survey research* and in data processing ana analysis* with emphasis on the use of computers and other mechanical devices and computer programme pack- ages ; 5. to arrange for guidance and consultancy services for social scientists requiring assistance In data processing and ana- lysis* 6. to act as liaison between official data producing agencies and social scientists in order to lASSIST Mewsletter, Vol. 3, No. *» (Fall 1979) incorporate the research interests of social scientists in data gath- ering and tabulation plans of these agencies? and 7. to establish collabora- tive relations with social science data archives abroad. To this was adaed in 1976 the function of compiling a National Register of Social Scientists in India to be revised per ioo i ca I ly . DAIA ACQUISITION AND OiSSEMI NATION The question of types of data to be acquired by the Data Archives and the sources of these data were first taken up. Three main sources of data were identified: (a) data generated through the ICSSR-funded projects; (b) data generated by the official and semi-official agen- cies; end (c) data generated by scholars working in the research Institutions and university depart- ments. Again* keeping In line with the general objectives of the Coun- cllt 1t was decided to acquire both survey as well as aggregate data relevant to the research needs of all the disciplines falling under the rubric of social sciences. However* in view of limited resources* both financial and oth- erwise* the Council laid down cer- tain priorities and gave the first priority to the acquisition of data sets generated by its own funded projects. The Council being one of the largest single social science research funding bodies in the country haa at the time of the establishment of the Data Archives* funded trore than 300 projects of which nearly 5 per cent had been completec. Since then* the number of the ICSSR funded projects have more than doubleo. It is* there- fore* quite under s t anoab I e that the Data Archives established by the Council would have a regular pro- gramme of acquiring and preserving data sets generated by its own funded or o ject s . During the past four to five years about 55 data sets have been acquired by the Data Archives. This also includes about ten data sets received from governmental agencies and other researchers who did not receive funds from the ICSSR. By any standard* this per- formance Is not quite satisfactory though understandable. It should be appreciated that the pace of developments* which appear to have resulted in the growth of such an activity else- where* took a late start in India and has been rather slow. The tradition of quantitative research* based particularly on secondary data analysis* is relatively new. A very small number of researchers make use of the facilities of com- puter technology and other mechani- cal devices for recording* process- ing and analysis of data. The availability of the hardware and software facilities for social sci- ence data processing are quite Inadequate. .hat Is worse* even the meagre facilities available are not being fully utilized because the community of social scientists is not oriented to and trained in the use of these facilities. Con- sequently* the majority of the scholars continue to analyse the data through hand tabulation and the raw data collected through the field sjrvey are rarely transferred on punched cares. Therefore* only a limitec number of useful machine-readable data sets are available for acquisition. lASSIST Newsletter, Vol. 3» No. 't (Fall 1979) General L tioned abo respons 1b I e acqulsl t Ion case of da mental agen adequate re and otherwl buted to acquis It Ion large numbe dest such Survey Org Bank of Ind tal agencle magnitude o data are social scl always aval ch they cou by soc I a I sc fore* extr Impossible t t1on with meani ngf u I L and to ma Interested y speak ve have for t to dat ta gene cles » sources set h the slo • For, r of g as t th anizat 1 1a« and s have f Impor ext reme ence r lable 1 Id be lentlst emel y d for an llmlte y orga ke th scholar i ng , i a been he slo a sets rated however f both ave als w pace over t overnme e Natio on, th other genera tant d L y rel esea rch n a f easily s. It if f Icul y s1ngl d res n1 ze t em ava s for r c t or pri w ra by g , I fin c of he y ntal nal e R gove ted ata< evan bu rm 1 ut Is, t, e In ourc hese 1 lab e-us s men- ma r 11 y te of In the ve rn- ack of anc lal ont r1- data ears a agen- Sample e serve rnmen- a vast The t for t not n whl- 1 I Ized t here- 1f not St Itu- es to data le to C grad an c los data coun tack of lute ma jo cles nece Ing f era Simu sc le tlve shou the of t a p the Thes inco of onsequ ua I ly all e coop gene try 1s le thi the op ly nee r offi the ssa ry their bly 1 Itaneo nt 1 St s s of Id wor nature hese d hased most e gro rporat social entl comi ut erat rati abs s ve Inio essa claL urge ar ra own n ma us ly as va k t ata prog 1m ups e t he r scl ent 1 y« ng f ron Ion ng oLut X Ing n t r y t dat nt n ngem dat chin t a wel r iou oget o lum and r amm port sh the t D th tal of t agenc e I y prob hat 1m a pro eed ent s a hoi e-rea grou 1 as s o her a e an help e f o ant ou Id esear st s Cou e V atta he 1 les esse lem . it i pres dud to m for ding dabl p o rep rgan nd d th In r a data als ch 1 In t ncil is lew that ck with mportant In the ntlal to It is s abso- s on all ng agen- ake the preserv- s , pre- e f o rm . f social resenta- i zat ions Identify e format evolving CQU i r Ing sets, o help nt erest s he data coll thes tion sigh ac qu usef of f the of t ever prog dire e c t i on e agenc s to t. T isition u I data actors, cont rol he acti , help ramme, ctly w1 and t abu L ie s . No these pr he entire and d 1 will dep not nee of the vi ties wh in st re direct I U be dis ation plans of immediate solu- oblems are in programme of ssemlnatlon of end on a number essarlly within Counc 11. Some ich would, how- ngthening this y or not so cussed now. INSTITUTION BASED £AIA ARgHjVES F Coun izat in I f eas rese sity have Host have the the fad in t In t info sy st I ar s tion tact Appr less fund se le deve seal rom cil 1 on ndl a ible arch dep g of the pres abs litl he he m r mat emat out or s eci a R de s c ted Lop the held of d Is 1n artm ener the ne erva ence es a r oce a jor 1 on leal side bey f a ting cide ana ins dat very the V ata ar nelth For, stitut ents, ated se ins cessar 1 1 on of vast ss of ity of about ly av a pa ond t n ind thes d to ot he tituti a arch begl lew t Chi va er n a lar ions ove r impor titut y f ac f the prop amoun deca the these ailab rt 1cu he pe 1 vi au e pro pr ov i r s ons t i ves nn 1 n hat I ac eces ge n and th tant ions Hit se d er t of y an case dat le t lar rson al b lem de n erv i o h on g» cent 1 1v1 sary umbe uni e ye d do les ata. sto dat d de s, a Is o s Inst al s cho s , eces ces elp a s the ral- ties nor r of ver- ar s, ata. not for In rage a Is ath. even not cho- itu- con- Lar. the sary to them mall The Inadequate financial resources of the ICSSR have been the main handicap for the develop- ment of this programme. These dif- ficulties are likely to continue for quite some time to come. Vast amount of financial resources would be needed to help these institu- tions purchase equipment needed for lASSIST Newsletter, Vol. 3, Nio. t (FaU 1979) the data organization and Young teachers and Ph.D. schoLars dissemination. Additional funds, in social science disciplines are on a regular basis would also be invited to participate in these necessa ry f or carrying out other courses of 3 t o A weeks duration, oata archival functions. It may These courses have proved to be not, therefore, oe possible for the quite useful and are expected to ICSSR to provide adequate funds for continue, setting up full-fledged institution based data archives. However, the Council may consider providing at least some funds to selectea insti- Gyj.DANC: AND CONSUklANCY SERVICES tutions having a large volume of good data for upaating and organiz- Quite closely associated with Ing data and for preparing the the programme of training courses, necessary documentation. Once this the Data Archives Initiated in Is done, one set of these materials 1974-75 a programme for providing may be made available to the ICSSR guidance and consultancy services Data Archives which will be made to the social scientists to tackle responsible for the data dissemina- their problems in data recoroing, tion task. Gradually, these insti- processing and analysis with the tutions can be developed to take help of mechanical devices. In over all the data archival func- order to make these services ava1l- tions. able to social scientists near their normal places of work, a num- ber of Institutions have been involved In this programme. Cur- lEiililNG CC2UESES rently, these facilities are avail- able, besides at the ICSSR Data As statea earlier, the overall Archives through (1) the Indian success of the data arthival pro- Institute of Management, Calcutta; gramme Is closely Llnkeo with the (2) the Centre for the Study of growth of quantitative techniques Developing Societies, Delhi; (3) in social science research in gen- the Sardar Patel Institute of Eco- eral and use of mechanical devices nomic and Social Research, Ahmeda- in data processing in particular. Daoi ('t) the Centre for Development These developments will generally Studies, Trivandrum; (5) the Gok- depend upon the overall system of hale Institute of Politics and Eco- higher education In the country. nomics, PoonaJ and (6) the Tata The structure of university educa- Institute of Social Sciences, Bom- t1on will take some time to change bay. The Scheme Is yet to gain and respond to these needs. In the momentum at all the centres, meantime, the ICSSR is trying to fill this void, to some extent, by organizing training courses in research -methodology and survey COLLiBOR ATIJ^E ELLAIlfiHS Kilt! fi*I* research techniques. Keeping in ARCHIVES ABROAD line with these objectives, the Data Archives decided to organize It would be readily admitted training courses in the application that the data archival programme of computer technology and other cannot be ceveloped In isolation of mechanical devices in social sci- the developments In this field ence research with emphasis on use abroad. It has been, therefore, of computer programme packages In decided that the Data Archives oata processing and analysis. - 10 - lASSIST Newsletter, Vol. 3, Mo. ^ (Fall 1979) should maintain collaborative publish uDdatea information period- relations with similar data ically. archives abroad for the exchange of Information* data» software and even data archival staff. For var- ious reasons, not much progress has DATA INF^RMATiON SERVICES been made In this field. All these functions discussed here are quite Important and are proposed to be continued and NATIO NAL BE'^^STER OF SQ£I^L further strengthened and expanded SCIEN TISTS during the course of the next few years. However* these alone will In the beginning of 1976» the not be enough. The scope of the Data Archives took up the task of Data Archives needs to be further co«p1l1ng a National Register of broadened for providing better ser- Soclal Scientists in India. It was vices. Systematic steps are neces- also decided that this Register sary to build a data Information should be revised periodically. system so that It can provide This attempt is aimed at filling referral services in the sources of the void In the area of basic and social science data In addition to comprehensive Information about the the existing programme of physical background* research Interests and acoulsition and dissemination of contributions made by the social important data sets. Making rele- sclence community in India. Ini- vant * clean and properly documented tlally* It was proposed to Include data available to the interested In this Register all the social scholar is an Important task no scientists in university depart- doubt. And* no less important is ments* colleges* research institu- it to provide Information on the tlons* governmental organization. availability of relevant data to a and private industry ana business scholar* even if the Data Archives and to cover Anthropology* Com- might not be In a position to ser- merce* Demography* Economics* Edu- vice that particular data to a cation* Geography* History* Inter- researcher at a particular point of national Relations* Linguistics* time. Management* Political Science* Psy- chology* Public Administration* In order to systematically build Sociology (including Criminology)* a meaningful data base for these Social Work* Communication (includ- services two programmes are pro- ing Mass Communication and Journal- posed to be initiated. First, we ism) and Law. However* because of plan to prepare an Inventory of practical difficulties the project current and recently completed had to be limited to the coverage social science researches in India of university departments* colleges and keep this information up to and research institutions. The date. Specifically* we Intend to first volume of this Register com- collect Information on the status prising about 7*000 social scien- of current researches* type of data tists* information regarding whom utilized* method of data collec- was collected through a mailed tion* processing and analysis, questionnaire (covering the period Efforts will be made to cover all up to December 1977) will be ready researches whether funded by the for publication soon. It is pro- ICSSR or not. Second* we propose posed to keep this information up to initiate a series of projects' to date* extend the coverage and lASSIST Newsletter, Vol. 3» No. t (Fall 1979) for preparing inventories on the Initially, the Data Archives had type of data, periodicity of col- decided to develop this software on lection publication of data and Its own. However, not much coulo unit of observation, etc., of be achieved in this regard for var- soclal science data generated by ious reasons. The most important official agencies. These two basic reasons being non-availability of sources will throw open such a vast adequate funds and skilled man- reservoir of information that when power. Even if the funds were to properly classified and organized, be made available, the problem of a would prove tremendously useful in non-ava1 I abl 1 1 1 y of properly discharging this function. More- trained programmers, who could overt this system would also help meaningfully Interact with social the data archives In Identifying scientists, understand their Important data sets at the appro- requirements and develop the soft- prlate time and facilitate their ware accordingly « will continue to acquisition. persist. Therefore, this programme will have to be developed over a long time. In the meanwhile, a beginning in this direction could B^y^^OPM^NT 2F SSEIUARE FACItiXX£§ be made by preparing an inventory of existing social science computer As has been emphasized earlier, programmes 1n the country and mak- all the data archives are supposed 1ng this information available to to acquire machine-readable data the Interested scholars. Such an and to organize them in such a form Inventory would, decidedly, Iden- as would make the data retrieval tify gaps in the availability of and dissemination least cumbersone these facilities, both cross-sec- and time consuming. In other tlonally and in different problem words, data archives would gener- areas. Once such Information ally acquire raw data transcribed beccnes available, the gaps that either on punched cards, magnetic may exist, could be plugged in a tapes or any other mechanical dev- phased programme under given prior- Ice. At the archives, the data Ities. normally have to pass certain 'acid tests', such as checks for wild In sum, 1t may be recapitulated codes, inconsistencies In coding that the initial period has been a punching, standardization of data period of mixed experience. The formats and code categories, etc., achievements during this period, before they become ready for disse- however, outweigh failures despite mlnation. For all these tasks, a no prior experience In this field data archive needs relevant compu- In the country. This period has ter programmes (software). This been quite significant in many software is used for data cleaning, ways. During the period the neces- transformation, organization and sary 1nf ras tuct ur e has been built, retrieval.' This type of software some of the programmes put on the is, therefore, necessary for the ground and sufficient experience ICSSR Data Archives. In addition, gained to help take a leap forward the Data Archives has to develop during the course of the next few and maintain another type of soft- years, ware if It proposes to cater to the statistical data processing needs However, it may be reiterated of the social science community in that the success of the data archi- the country, val programme in the country would depend on the proper development of - 12 - I ASSIST News let t er , Vol No. t (Fall 1979 ) var i Prop st re vent sage All the for gram t1ta in s deve come the In ment of t sure is 1 ous er s ng t he 1 onal d ea this sing the me is tiwe oc ial lopme abou uni ve the c s» s he Da to c n ant functions out teps are n n these func or unconvent rlier or new is Important le most impo development an extensive techniques a science rese nts are prima t through th rs i t y system ountry. Th important f ta Archives* ome about* icipation of I ined e c es s t i ons i ona I I y p rtant of th use nd au arch, rily e ch of e ese or t h are In ef thes above . a ry to : con- « env i - roposed . And y e t .t factor is pro- of quan- t omat ion These going to anges in ducat ion develop- e growth slow but feet* it e devel- opments init iate gramme. quicken this pr initiate dance an training and res newly e the inve and so hopefull to this the leas the soci will det of devel India. that the C the dat However the pace ogrammet d its prog d consul t a courses i earch me nvisaged p nt ory of f t ware fa y » give a p rogr amme • t « it is t al science ermine t h opment of of d the r amm ncy n da thod rogr cur r cili ddit L he c cu e ra this i I dec r c h i va in or eve I op Counc est I i servi ta pro ology . ammes « ent r ties i ona I ast « oopera mmuni t t e an progr ided to I pro- der to ment of i I has ke gu i - ces and cess i ng The like esear ch would* support but not t ion of y which d level amme in - 13 -