next up previous contents
Next: Design of the GILDAS Up: IRAM Memo 2012-? Preparing Previous: Contents   Contents

Introduction

The size of the datasets produced by the (sub-)millimeter single-dish and interferometers, including the IRAM 30m, Plateau de Bure interferometers and ALMA, experience a tremendous increase (because of wide bandwidth receivers, spectrometers with thousands of channels, multi-beams, and/or new observing modes like the interferometric on-the-fly).

In the IRAM 30m context, the new FTS backend delivered during 2011 can produce spectra of up to 37275 channels. Combined with On-The-Fly map mode which can contain 10,000's of spectra, the number of individual data values can now easily reach 2 billions. The advent of NOEMA (a major upgrade of PdBI capabilities) will also lead to large data sets.

While CLIC and CLASS data formats have their own limitations (which will be described in another document), imaging through gridding and deconvolution uses the GILDAS data format (hypercubes and tables), which were initially implemented for 32-bit machines. The original implementation of this Gildas Data Format (hereafter GDFV1) is counting the number of (4-bytes) elements it contains in a signed integer value (Integer(4)), which, in IEEE arithmetics, is limited to $2^{31}-1 \simeq 2.10^9$ elements. Replacing this signed integer value (integer(4)) by a signed long value (integer(8)) would have not helped as the dimensions of the data array are also coded with a signed integer value. The only correct solution was to also store these dimensions as signed long values with effect everywhere in the GILDAS code (both kernel and packages). This way 1D arrays can now contain as much as $2^{63}-1 \simeq 9.10^{18}$ elements.

In addition to these size limitations, new observing modes, such as polarization or interferometric On-The-Fly, becomes available. Many new requirements appeared to support these modes: Support of more than 4 dimensions, new header parameters, more flexible UV tables, ... Time had come to revise the Gildas Data Format to a new version (hereafter GDFV2).

Section 2 summarizes the GDF design. Sections 3 and 4 describes how the GDF format can store two main kinds of data, namely hypercubes and tables. Changes for end-users and programmers are detailed respectively in Sections 5 and 6. Backward compability and other miscellaneous issues are dealt with in Section 7. For convenience, the data format and the public application program interface of the GIO library is cut and pasted from the code in Appendix A to D. Programmers are encouraged to look directly into the source codes for up-to-date information.


next up previous contents
Next: Design of the GILDAS Up: IRAM Memo 2012-? Preparing Previous: Contents   Contents
Gildas manager 2014-07-01