US Marine Meteorological Journals Collection (deck 704; 1878-94) (by ESRL)

1. Introduction

The US Marine Meteorological Journals form part of Record Group (RG) 27, (Records of the Weather Bureau, Records of Marine Observations, Entry 127) at the US National Archives and Records Administration (NARA). The MMJ were transferred to NARA from the Weather Bureau in 1936, described by an Identification Inventory documenting that accession as follows:

"Log Books containing extracts of logs of oceangoing ships from 1879 to 1893. Accompanying the log books are three boxes containing a card index of the books, each card giving the name of the ship, master, route, dates en route, and the number assigned to the book. The log books were received from the U.S. Hydrographic Office about thirty years ago when the meteorological work of that office was transferred to the Weather Bureau. The log books were kept by the Master or another official designated by him and the information was sent to the Hydrographic Office where it was used for making up pilot charts and studying weather conditions. It is interesting to note that for a long period these books were sent to Lieutenant M.F. Maury, 'Pathfinder of the Seas,' at the Naval Observatory."

A Location Register obtained from NARA during a visit in 1989 referred to Entry 127 as containing 1,955 volumes (135 feet) from 1879-1893. During that visit, journal No. 2 was examined, resulting in the following description of the format:

"Leather-bound, 3/4" thick, landscape-orientation, 11.5" x 9" volume... A lengthy introduction section, including example journal pages with fictitious data printed in red.... Journal filled with genuine 2-hourly observations.... Stamped in red with barometer correction and 'Records transcribed at Pittsburgh, PA, 1933-34,' and with additional data interpretation marks in red pencil."

This report provides additional details about the MMJ collection, gleaned from the 1989 visit, plus two subsequent visits in 1993 and 1997, to NARA. In 1998, agreement was reached between NARA and NCDC to microfilm the collection, as the initial step towards digitization of the marine meteorological observations.

In May 1998 a data exchange agreement was reached between the US and China, for China to digitize the MMJ collection, using paper copies created from the microfilm by NCDC. This document describes the digitization format, and this report from China's National Marine Data and Information Service (NMDIS) describes completion in late 2000 of the digitization and quality controls that were applied by China to the data. The digitized data were provided to NOAA's then-Climate Diagnostics Center (now within the Physical Science Division of the Earth System Research Laboratory) for conversion and blending into ICOADS.

The microfilm was eventually also converted into TIF image format by NOAA's Climate Database Modernization Program (CDMP), and is available from CDMP's archive (186,162 images). It was discovered later that 31 Journals were inadvertently misfiled at NARA among the US Maury Collection, and which have not yet been microfilmed or digitized. These missing Journals, listed here, are estimated to contain approximately 28K observations from 1879-90. Future work has been proposed to CDMP to image and digitize the missing Journals. CDMP work is also planned to make digitally available the card index to the complete collection (Entry 128).

Within the collection, three distinct editions of the Journals, and of the accompanying Instructions, have been located (determining whether other editions exist would require examining every Journal). The column structure (Table 1) for recording the data is the same in the August 1878 and February 1883 editions, but there are differences in the Instructions. The November 1884 edition marks an important change in the data, with the addition of columns for compass correction, true course, and true wind direction.

Table 1. The structure of columns used for recording the data in the three known Journal editions. The November 1884 edition transitions to 20 columns (adding: compass correction, ship's true course, and true wind direction) but the Instructions still group these under only 17 headings (as numbered in the instructions). The (year and month) dates of the editions are from p. 17 of the instructions (the Journal title page also contains a year, which is frequently later than that on page 17 and does not appear to be useful for dating the edition). PDFs of selected high-quality examples of the three editions can be downloaded from here.

2. Translation results

The MMJ collection as digitized by China has now been translated into the Long Marine Report (LMR) and International Maritime Meteorological Archive (IMMA) formats (see specifications), for blending into ICOADS. The actual period of record of the digitized data was discovered to be 1878-94, with 1.76M resultant IMMA reports (58,562 reports, 3%, were rejected from the translation because latitude and longitude were not reported or could not be computed). Figure 1 shows the temporal coverage, and Figure 2 shows the overall, and annual, spatial coverage of the collection.

Figure 3 shows how the different variables were assessed by quality control (QC; "trimming"). The grey regions in the plot thus indicate that sea level pressure (SLP) is the best-sampled variable, followed by wind, sea surface and air temperatures (SST and AT), and relative humidity (RH), respectively. In addition, SLP has larger numbers of data flagged as suspicious by the trimming procedure than the temperatures and wind, whereas the trimming limits are almost entirely missing for RH.

Figure 4 shows the distribution of SLP and other variables with latitude, also indicating whether the data were within or outside the broadest (4.5σ) trimming thresholds (or that the limits were missing). The plot of SLP reveals some suspicious patterns of biased data, in agreement with the relatively high trimming rates shown in Figure 3; and the plots of wet bulb and dew point temperatures (WBT and DPT) show the impact of the missing trimming limits for RH (since those humidity variables are trimmed on the basis of the RH flag).

Figure 5 shows the availability of ship's course and wind direction information in the collection, clearly reflecting the extension of the Journal's structure after the November 1884 edition to include true course, true wind direction, and compass correction (ref. Table 1).

Figure 6 shows a ship's track before and after magnetic correction.

Figure 1. Temporal coverage of the MMJ collection, shown as thousands of reports per year (output IMMA). The bars are colored differently (in 2-year blocks) to correspond with Figure 2.

Figure 3. For the 1.76M reports in the MMJ collection, the first bar plot shows the numbers of reports made at local nighttime (ND=1) versus daytime (ND=2). The remaining bars show the values of the "trimming" (QC) flags for different variables:

SF:  sea surface temperature flag
AF:  air temperature flag
UF:  U wind flag
VF:  V wind flag
PF:  sea level pressure (SLP) flag
RF:  relative humidity flag

Table 2. Description and definition of the trimming flag values, where a_i is the individual observation under scrutiny, g is the smoothed median, and σ₁ and σ₅ are the smoothed lower and upper median deviation (values 8-A are currently unused). See the electronic documentation <stat_trim> for additional information.

3. Unresolved Issues

• The general absence of trimming limits for RH during the time period of the collection (see Figures 3 and 4) represents a significant problem for the calculation of ICOADS 2° trimmed monthly summaries. Those climatological limits were calculated based on the original COADS Release 1 data (issued in 1985), in which very few humidity data appeared (i.e., during the earliest trimming period, 1854-1909, that was defined for Release 1). One interim solution that could be considered would be to use the RH limits for the next most recent trimming period (1910-49), as QC basis for these early humidity data.
• Reel numbers (referring to the microfilm copy of the collection) and Journal numbers (referring to the original Journal numbering system) are stored both in the "index records" associated with the CDMP image collection, and in the digital data. In these fields, some errors are known to exist, as well as some unresolved inconsistencies between the index records and the digital data, which it may be possible to correct or improve through future work. In addition, the ship names as separately recorded in the index records and in the digital data contain variations and inconsistencies, which it might also be beneficial to edit for greater uniformity (e.g., to better support ship track checking or linkage with external ship metadata).
• As discussed in sec. 2, 3% of the data (about 59K reports) were rejected from the translation because latitude and longitude were not reported or could not be computed. The calculation of longitude and latitude was basically limited to location information available on a given calendar day (i.e., one Journal page). The possibility of interpolation of positions across days could be considered in the future, if the collection was reprocessed, as a means of recovering some of the rejected observations.

[Documentation and Software][Translation information]