GEO 945 ANNALS OF GEOPHYSICS, SUPPLEMENT TO VOL. 47, N. 2/3, 2004 2 Nowcasting, forecasting and warning for ionospheric propagation: supporting databases RICHARD STAMPER (1), JEAN LILENSTEN (2) and NORBERT JAKOWSKI (3) (1) Rutherford Appleton Laboratory, Chilton, Didcot, Oxon, U.K. (2) Laboratoire de Planétologie de Grenoble, Bâtiment D de Physique, Grenoble, France (3) Deutsches Zentrum für Luft und Raumfahrt (DLR), Institut für Kommunikation und Navigation (IKN), Neustrelitz, Germany The use of data is essential in the context of nowcasting, forecasting and warning of ionospheric propagation conditions, with roles to play in the development, evaluation and operation of models and services. Descriptions are given of three databases that have been established in the course of the COST 271 Action: a database of prompt ionospheric soundings, an extension to a database generat- ed by the EISCAT incoherent scatter radars, and a database intended to facilitate evaluation of TEC estimation methods. Each database includes some background information, a description of the con- tents and interface, and instructions as to how to gain access to it. 2.1. INTRODUCTION The Memorandum of Understanding of the COST 271 Action and the terms of reference of Working Group 1 make clear the importance of data for the work of the Action in general and for the specific fo- cus of the work package on nowcasting, forecasting and warning. In particular, there are commitments to collect additional and new ionospheric and plasmaspheric data for now-casting and forecasting purposes, and to establish a space-weather database consisting of both past and new measurements. Such commitments are clearly proper when one considers the roles that observational data play in the context of the investigations undertaken during the Action into the propagation environment and communication systems. Three key roles can be identified: – When developing a model, data provide the framework in which the model is specified and parametrised. – Once a model has been specified it needs thorough validation against the full range of condi- tions it might encounter. – An operational nowcasting or forecasting service will inevitably require data describing the past or current conditions in order to generate its outputs. Data have always been used in these ways but the importance of databases is that they simplify ac- cess to data for which access would otherwise be difficult. This could be because the data are original- ly dispersed across multiple sites or because they are in multiple or unfamiliar formats. If anything, these problems have only been multiplied by the great diversity of data sources now accessible on the Internet. This paper presents information on three databases developed as part of the COST 271 Action: 946 Richard Stamper, Jean Lilensten and Norbert Jakowski – Prompt Ionospheric Soundings at RAL, Chilton; – EISCAT database extension at CDPP, Grenoble; – TEC Evaluation DAtabase at DLR, Neustrelitz. Each of these databases makes access to a wide body of data available via a single source, and presents that data in standardised and readily comprehensible formats. 2.2. PROMPT IONOSPHERIC SOUNDINGS DATABASE 2.2.1. Background and motivation The original rationale for compiling the database was the need to understand better how the regional and global ionosphere responds to the impact of space weather events. Ionospheric databases have hith- erto concentrated on scaled parameters, but to advance our understanding of ionospheric propagation ef- fects more detail is needed. The new database accordingly stores full ionograms and layer traces. This enables the construction of bottom-side ionospheric profiles, and also provides a large of body of data ca- pable of supporting future developments in nowcasting and forecasting of propagation conditions. The database is located at the Rutherford Appleton Laboratory (RAL), where it has been developed as an extension of the existing database system of theWorld Data Centre that archives vertical sound- ings from the ionosondes run by RAL at Chilton in the U.K. and Stanley in the Falkland Islands. The work to extend this system began in April 2002, funded by a grant from the U.K. Radiocommunica- tions Agency, and it now accepts prompt ionospheric vertical soundings from across Europe. Note that although the database is designed to accept and make available data in near-real-time, the accumulated historical data are still valuable. Any operational nowcasting or forecasting system will necessarily need to use prompt data as input. It is therefore of little practical use to develop such systems using only high quality, manually scaled ionospheric soundings, since such scientific quali- ty data is rarely available in real-time. It is therefore essential for such tools to use a representative set of historical real-time data when being developed and tested, that is, data that were at one time prompt and have not been adjusted manually since. The RAL database provides just such a resource. 2.2.2. Database content The database takes data from instruments across Europe, currently all DPS or DGS sounders man- ufactured by the University of Massachusetts, Lowell, Center for Atmospheric Research (UMLCAR), and known generically as ‘digisondes’. The full list of participating instruments and the institutions that operate them at the time of writing is – Chilton, U.K. (RAL); – Tromsø, Norway (University of Tromsø, QinetiQ); – Juliusruh, Germany (IAP Kühlungsborn); – Athens, Greece (NAO); – Rome, Italy (INGV); – El Arenosillo, Spain (INTA); – Pruhonice, Czech Republic (Academy of Sciences); – Ebro, Spain (Observatorio del Ebro). The standard output files from these sounders are sent to or retrieved by the RAL computers within a few minutes of the soundings being carried out. The files are catalogued and archived automatically on arrival. The UMLCAR digisondes are supplied with a sophisticated software package called ARTIST that carries out automatic scaling of the ionograms. The files delivered to the database therefore contain not 947 Nowcasting, forecasting and warning for ionospheric propagation: supporting databases only the ionogram, but also auto-scaled traces and autoscaled parameters. A standard subset of the auto- scaled parameters are extracted and stored separately from the raw ionogram files for ease of access. Because the initial set of instruments supplying data have all been digisondes, the UMLCAR SAO file format has been adopted as the standard archive format for storing the parameters and traces from each ionogram. The format is sufficiently generic that it should be possible to re-encode the native output from any other type of sounder into the SAO format for archiving. 2.2.3. User interface and data products There is public access to the database through the Web link http://www.wdc.rl.ac.uk/cgi-bin/di- gisondes/cost_database.pl/. Users are required to register so that they have a defined user area for storing temporary files, but the registration procedure is automatic. The front page is shown in fig. 2.1. It consists of a form which allows the user to select data by station (via a selection list or by clicking on the map of Europe) and by arbitrary time intervals; one can choose to download the raw data files and ionograms, or one can display or download a number of other data products. For each of the contributing instruments the database holds some background information. This can be retrieved by choosing the STATION DETAILS option from the web form on the front page. An example of the output of such a request is shown in fig. 2.2. For each instrument there is displayed its URSI code, its physical location, details of the principal contact person and in- Fig. 2.1. The front page of the Prompt Ionospheric Soundings Database, showing the options for selecting da- ta by station and date, and for selecting the type and mode of delivery of the data product. Fig. 2.2. The details of stations available from Ionospheric Soundings Database. 2.22.1 948 Richard Stamper, Jean Lilensten and Norbert Jakowski Fig. 2.3. Displaying availability of data files in the Prompt Ionospheric Soundings Database. Fig. 2.4. Listing auto-scaled parameters in the Prompt Ionospheric Soundings Database. formation about the institution responsible for running the instrument. Wherever possible, links are given to appropriate e-mail addresses and web sites. For example, in fig. 2.2 the section for the sounder at Athens has links to the web page for the instrument (the sounder name ‘Athens’ is the link), the e-mail address and personal web page of the principal contact, and the web site of the hosting institution. The other selection options allow data to be retrieved in a variety of forms: tables of data avail- ability, a table of auto-scaled parameters, line plots of parameters over time, true-height bottom-side profiles calculated using POLAN, and zip files of raw data files. For detailed data analysis the FILE DOWNLOAD option enables users to retrieve the raw data files. Suitable third-party software such as SAO-Explorer from UMLCAR is necessary to process these files. Before downloading data it is best for a user to check what files would be retrieved; this is done with the DATA AVAILABILITY option, for which an example query result is shown in fig. 2.3. To view all the auto-scaled parameters the user should select the Parameter Data option; this gener- ates a table listing 16 parameters standardly scaled by the ARTIST auto-scaling software on the digisonde sounders. Figure 2.4 illustrates the result of such an operation, where the table is sorted by station in this instance; it is equally possible to sort the results by time, where all times are UT. Alternatively, one can generate a simple line plot of the foF2, foE and f min auto-scaled parame- ters (see fig. 2.5), useful for getting a quick view of data availability and quality. The time interval covered by such a plot is set by the user when specifying the query, which makes it possible to focus on the detail of a particular period of interest or, alternatively, to get an overview of a long interval. Finally, it is possible to run the POLAN ionogram inversion algorithm on any auto-scaled traces. For each sounding the result is a list of (frequency, height) pairs, and a short extract of one such profile is shown in fig. 2.6. 2.3 2.4 949 Nowcasting, forecasting and warning for ionospheric propagation: supporting databases 2.2.4. Future developments There is work continuing to extend the number and geographic spread of contributing instruments. The user interface to the database will also continue to be developed to improve its flexibility and ease of use, with three additions currently planned: 1) Some elements of user control will be added to the plotting interface to permit users to select the set of parameters to plot, rather than being given a pre-defined set. 2) Access to the WDC holdings of definitive manually checked scaled parameters will be pro- vided through the same interface as the auto-scaled prompt parameters currently available. 3) A facility for setting up automatic transfers of data products will be added. By enabling users to specify a set of data products that should be transferred regularly by e-mail or FTP, this will support the use of the database by operational services that need regular prompt delivery of data inputs. 2.3. EISCAT DATABASE EXTENSION 2.3.1. Background and motivation In 2002, the International Telecommunication Union Radiocommunication group (ITU-R) issued a recommendation for a proposed data bank for trans-ionospheric model testing. The intention of the recommendation was to make the EISCAT radar data available to the wider community in a form that would allow testing of propagation models and, in particular, the TEC, hmF2, NmF2, and foF2 pa- rameters had to be easily readable and freely down-loadable. Actions to meet this recommendation Fig. 2.5. Plot of ionospheric parameters over along time interval from the Prompt Ionospheric Soundings- Database. Fig. 2.6. True-height analysis using POLAN from the Prompt Ionospheric Soundings Database. 2.5 2.6 950 Richard Stamper, Jean Lilensten and Norbert Jakowski have been undertaken by CDPP (Centre des Données de la Physique des Plasmas), a data archiving and distribution centre for Space Plasma Physics, implemented by the French National Centre for Scientific Research (CNRS) and the French space agency (CNES), which provides the French EIS- CAT database. 2.3.2. Incoherent scatter radars: EISCAT Incoherent Scatter (IS) radars are a powerful ground-based tool for the study of the Earth’s ionosphere and its interactions with the upper atmosphere, the magnetosphere and the inter- planetary medium. The ‘incoherent’ scatter echo is the result of the scattering of electromag- netic energy radiated from the radar by electrons in the ionospheric plasma which are them- selves controlled by the much slower and more massive positive ions. Most of the returned power is due to scattering from electron density fluctuations caused by the presence of these ions. The frequency spectrum of the received signal therefore provides information about the temperature, composition and velocity of the ion population. The incoherent scatter echo has a characteristic double-peaked ion-line spectrum which also conveys information about the tem- perature of the electrons themselves. Besides the main ‘ion-line’, the incoherent scatter spec- trum also contains other, usually weaker, components including two ‘plasma-’ or ‘electron- lines’ representing scattering processes where the electrons act as if the ions were absent. The EISCAT (European Incoherent SCATter) scientific association is an international re- search organisation operating three geophysical research incoherent scatter radar systems, an ionospheric heater and a research ionosonde (known as the Dynasonde) in Northern Scandi- navia. Two of the radar transmitters are located close to the city of Tromsø in Norway, while the third, the EISCAT Svalbard Radar (ESR), is located near Longyearbyen on the Svalbard archipelago. The work described here concentrated on data from the auroral region radar over Tromsø; this was obtained using one of the standard operating modes of the EISCAT radars (Rishbeth and van Eyken, 1993). The analysis of the scattered signal allows one to deduce the ion and electron temperatures, the ion velocity parallel to the magnetic field, and the electron density. Finally, one of the unique features of the EISCAT radar is that it has three widely sep- arated receiving stations, thus enabling the observation of the northward and eastward electric field components. 2.3.3. CDPP EISCAT database The CDPP hosts the French EISCAT database which archives all the ionospheric data from the radars that have been analysed in France since the radars first became operational in 1981, and the database can be accessed at http://www-eiscat.ujf-grenoble.fr/. The data can be retrieved ei- ther in graphical format as GIF plots or as simple ASCII text format tables. The data are processed with either a 5 min or 1 min integration time. The extension to the database has made use of data taken by the CP1 (Common Programme 1) experiments, in which the Tromsø antenna is kept fixed parallel to the magnetic field. The remote sites make continuous measurements at the F-region position (typically at 278.6 km al- titude). For the CP1 data the EISCAT signal is made up of three parts. The power received on the antenna gives a first estimate of the electron density with a spatial resolution of 3 km up to about 420 km. This measurement does not give the incoherent scatter spectrum and there- fore can only give access to the electron density. Until the year 2000, EISCAT used a single pulse in the F-region, and multipulses or alternating codes in the E-region. The multipulse cod- ing or alternating code have short time and space resolution (about 5 km) and make measure- 951 Nowcasting, forecasting and warning for ionospheric propagation: supporting databases ments of all parameters (electron density Ne, electron and ion temperatures Te and Ti, ion ve- locity) typically in the E- and lower F-regions (altitudes from 90 to 250 km). The single pulse coding is a longer pulse (350 µs) giving a coarser spatial resolution of about 30 km at higher altitudes (from about 140 to 600 km). It also allows retrieval of the four main parameters. 2.3.4. Extending the database To act on the ITU-R recommendation, the Grenoble EISCAT database was updated with the In- tegrated Total Electron Content (ITEC) over the altitude range from 90 km up to about 600 km, and the F2-region parameters (hmF2, NmF2, foF2). Since the best electron density resolution is obtained with the power profile, it was necessary to correct its estimate with the electron and ion temperatures measured by the other pulse schemes. This has been done by fitting the multipulse and single pulse temperature data with a spline function. If Ne(z) is the true electron density at altitude z, and Npp(z) is the power profile estimate, we have / N z T T N z 1 1 2 e D D e i pp 2 2 # = + + +a a ^ ^ ^ ^ h h h h (2.1) where αD is the debye length. Finally, in order to get the TEC estimate with data measured between 450 and 600 km, we adjusted the single pulse measurement and the corrected power profile with a cubic spline interpolation. This analysis scheme allowed processing of the whole database over more than 1 solar cycle. Fig. 2.7. Start page of the EISCAT database. Fig. 2.8. Selecting data from the EISCAT database. 2.7 2.8 952 Richard Stamper, Jean Lilensten and Norbert Jakowski Fig. 2.9. Plotting data from the EISCAT database. Fig. 2.10. Listing data from the EISCAT database. 2.3.5. User interface and database outputs The data is freely available via the web site http://www-eiscat.ujf-grenoble.fr/. Upon entering the database one is presented with the option to select quick-look plots or ASCII data listings by year (see fig. 2.7), after which one is taken through to a page (fig. 2.8) containing tables from which individ- ual plots or data listings can be selected. The form of the plot and listing outputs are shown in figs. 2.9 and 2.10 respectively. 2.4. TEC EVALUATION DATABASE (TECEDA) 2.4.1. Background and motivation During the COST 271 several projects dealt with estimation of TEC using a variety of differ- ent techniques. These activities can also be put in the broader context of the interest of the Ionos- phere Working Group of the International GPS Service (IGS) in evaluations of global TEC maps, with plans to release such evaluations as an official product of the IGS in the near future. The COST 271 countries are well placed to contribute to TEC validation; they operate a num- ber of ionospheric sounding facilities, are active in ionospheric modelling and have started to de- velop multi-sensor assimilation techniques for comparative studies. It was therefore decided to es- tablish a database of independent measurements of ionospheric properties that could be used as the basis for systematic comparative studies between different data sets to derive conclusions about the achieved accuracy of TEC estimates and the potential of modeling and assimilation work. It was further decided to concentrate on a specific period, chosen to coincide with the HIRAC high-resolution GPS campaign which ran between 23rd and 29th April 2001. 2.9 2.10 953 Nowcasting, forecasting and warning for ionospheric propagation: supporting databases 2.4.2. Database contents The database is organised to accommodate data of various types and origins in a single scheme, so that data of interest can be easily located and retrieved. The data holdings are presented as a di- rectory hierarchy on an FTP site. The top level directory contains subdirectories named after the lo- cation to which the data relates: examples include ‘Athens’ and ‘Chilton’ referring to the location of ionosondes, ‘U.K.-TOMO’ for a U.K. tomographic receiver chain, and ‘IRO-CHAMP’ for radio oc- cultation data taken by the CHAMP spacecraft. Within each of these location directories are three subdirectories: /tec /pro /rawdata The purpose of these subdirectories is to hold files containing, respectively, derived TEC estimates, electron density profiles, and raw data. A file naming scheme applies to the files in the TEC and pro- file subdirectories; each file has a name of the form bbb-ttt+sssss-ooooo_yyyy_ddd where bbb is a 3-character code indicating the base type, namely ‘TEC’ or ‘PRO’ Fig. 2.11. Vertical electron density profile from the CHAMP space craft, extracted from the TECEDA database. Fig. 2.12. TEC time series from NIMS satellite transmissions, extracted from the TECEDA database. 2.122.11 954 Richard Stamper, Jean Lilensten and Norbert Jakowski ttt is a 3-character code indicating the sub-type of the base type to give the derivation of the data, drawn from the set ASS Assimilation EIS EISCAT radar data IVS Ionospheric Vertical Sounding MOD Model output TOM Tomography sssss is a 5-character code to represent the receiving station ooooo is a 5-character code to represent the organisation or facility which has created the data yyyy is a 4-digit year ddd is a 3-digit day-of-year (January 1st = 001). An example of such a name is PRO-MOD+AT138-dlrnz_2001_101, which would contain ionos- pheric profiles (PRO) produced by a model (MOD) for the location of the Athens ionosonde (AT138), generated by the DLR at Neustrelitz (dlrnz) for day 101 of 2001 (11 April 2001). Wherever possible these TEC and profile files are presented in a straightforward human-readable and self-describing ASCII format. The hash character (#) is used to indicate comment lines which are used to create a file header which includes the name of at least one contact person. The header is terminated by a line reading ‘# end’. The data follows this header and is typically arranged in columnar format. Two examples of parts of such files are presented in figs. 2.11 and 2.12, showing respectively a vertical electron density profile and a reconstructed TEC series. As stated previously, one of the motivations for creating the database was the HIRAC campaign. Consequently, the data is primarily concentrated on the interval 23-29 April 2001, with time series data at 10 min resolution wherever possible. Data from other periods can usually be obtained by contacting the data provider, as detailed in the data file headers. Where data are held in some non-ASCII or non-tabular format, the data providers should have supplemented the data files with a text file giving sufficient supporting information to make the data files usable by the general user. 2.4.3. Database access The data are available from an FTP site hosted by the DLR, Neustrelitz, and access is restricted by username and password. Anyone wishing to have access to the database should contact Dr. Nor- bert Jakowski (Norbert.Jakowski@dlr.de) and details of the host computer and account details will be provided. There is also a general invitation to anyone with relevant data, particularly for the inter- val of the HIRAC campaign, to deposit data in the database. In particular, for access to more ionos- pheric radio occultation data obtained on-board the CHAMP spacecraft (Jakowski et al., 2002), one should contact the Information and Science Data Center (ISDC) Potsdam (http://isdc.gfc-pots- dam.de/champ/welcome.html) or DLR directly (Norbert.Jakowski@dlr.de). DLR routinely feeds the ISDC with ionospheric radio occultation data products. 2.5. CONCLUSIONS During the course of the COST 271 Action significant new data resources have been established, and the three principal examples have been described. The contents, intended user communities, and key features of these three database are different but complementary: – The database of prompt ionospheric soundings at RAL is intended to draw together ionospher- ic vertical sounding data that would otherwise be dispersed across multiple data providers. A key fea- ture of this database is that the data are ingested promptly so that the database can be used as a re- source for the development and operation of nowcasting and forecasting tools. 955 Nowcasting, forecasting and warning for ionospheric propagation: supporting databases – The EISCAT database extension at CDPP is intended to broaden the community of users that can make use of the powerful research tool of the EISCAT radars, by presenting the EISCAT data in terms of parameters commonly used by the radiocommunications community. – The TECEDA database is intended to gather together a wide variety of related data for a spe- cific time period. This facilitates detailed study and comparisons of the properties and behaviour of the ionosphere and propagation conditions during this interval. All three databases are resources with many actual and potential users, and represent one of the enduring legacies of the COST 271 Action. It is hoped that they will be of use to a global communi- ty of users, not just those in the European area, and their wide use is encouraged. ACKNOWLEDGEMENTS EISCAT is an international association supported by the research councils of Finland (SA), France (CNRS), the Federal Republic of Germany (MPG), Japan (NIPR), Norway (NFR), Sweden (VR) and the United Kingdom (PPARC). The prompt ionospheric database at RAL was funded by the Radiocommunications Agency, now part of OFCOM. Thanks must go to Chantal Lathuillère, Béatrice Pibaret and Bénédicte Lemieux-Dudon for their help in implementing the extensions to the Grenoble EISCAT database. REFERENCES JAKOWSKI, N., A. WEHRENPFENNIG, S. HEISE, C. REIGBER and H. LUEHR (2002): GPS radio occulta- tion measurements of the ionosphere on CHAMP: early results, Geophys. Res. Lett., 29 (10), doi:10/1029/2001GL014364. RISHBETH, H. and A.P. VAN EYKEN (1993): EISCAT: early history and the first ten years of operation, J. Atmos. Terr. Phys., 55, 525-542.