![]()
| ||||||
ICOADS Web information page (Wednesday, 25-Jul-2012 18:06:59 UTC): US Navy Hourlies (deck 117; 1951-64) (by ESRL)1. Introduction Deck 117 forms a significant (approximately 3M reports) and largely untapped data source for ICOADS, which (according to AWS-WB 1964) is an extension of deck 110 (US Navy Marine); however, the original card deck formats for 110 (AWS-WB 1958) and 117 are very different. Three major versions of the deck 117 data have been located and converted into IMMA format, which can be distinguished by source ID (SID): SID = 140: US Navy Hourlies: Deck 117 in TD-1100 format (2,506,125 input reports; 2,503,540 output reports): abbreviated as "TD1100" belowThe output is less than the input in each case due to the removal of exact duplicates, etc. SIDs 141 and 142 appear to be largely identical data (specifically 99.7881% of the FOSDIC and 99.798% of the DSI1117 reports were duplicates of each other). However, the DSI1117 data (NCDC 2003) were supplied as two separate files, one described as "navy-marine-sfc-obs" and the other as "osv-ice-island" (i.e. separating out the Ocean Station Vessel [OSV] and Ice Island data). Rather than assign another SID to data from the latter file, we have made different platform type (PT) assignments to those data as explained in Figs. 3-4. Additionally, 16K reports from deck 117 were included in COADS Release 1 (Slutz et al. 1985), with 11K retained after duplicate elimination processing, and thus form a fourth source for limited intercomparison (see Annex). We know that these data were supplied by NCDC in TD-1100 format for Release 1, because they came from the following larger TD-1100 sources (note: these deck 117 data appear to have been utilized for Release 1 primarily for Ocean Station Vessels): SID = 1: Atlas (20 reports)Also we note that some deck 117 records were tapped to fill gaps in Arctic Drift Stations including Fletcher's Ice Island (T-3) (ref. Table 1 and Figure 2 in Elms et al. 1993), but those data have not yet been fully blended into ICOADS. The original punched card deck format is documented by an initial reference manual (AWS-WB 1957), as well as by a revision of that manual (AWS-WB 1964); we used the revised version as the format reference. NCDC (2003) appears to be essentially a modernization of the information in AWS-WB (1964), omitting some information (e.g. as discussed in notes following Fig. 4 caption). The original card deck data were largely but apparently not entirely recovered by NCDC from "FOSDIC" film (Greenough 1998; documentation available at http://rda.ucar.edu/datasets/ds258.4/ shows 3.053M reports were once available according to an early punched card inventory). The TD1100 data were obtained from NCDC in 2009. While these data appear to be in the TD-1100 format in terms of the "Standard" and "Additional" character positions (1-93), it is important to note that deck 117 is not further documented within NCDC (1968) as being among the 17 decks originally considered part of TD-1100, and thus any "Conversion Procedures or Explanations" and "Supplemental" fields for deck 117 contained in TD-1100 format appear to be undocumented (and at this writing no additional information has been obtained from NCDC). ![]() Figure 1. Deck 117 coverage for the period of record 1951-64 with the FOSDIC (red) version overlaid by the TD1100 (gray) and the Release 1 (blue) versions (i.e. the "pink" color shows the overlap, such that TD1100 is almost 100% duplicated in FOSDIC). There were no TD1100 data exactly on the 1-degree box straddling the dateline. The missing TD1100 data at 90-99°E/W longitudes were invalidly located at 190-199°E/W (those reports were therefore considered erroneous and not output from this translation, but could potentially be recovered via a longitude correction). Apparently landlocked reports were previously edited out of the TD1100 data. ![]() Figure 2. Number of reports per month in the DSI1117 (indigo), FOSDIC (red), TD1100 (gray), and Release 1 (blue) versions. ![]() Figure 3. Spatial distribution of the Ocean Station Vessel (OSV) and ice island component of DSI1117, with individual ships plotted separately using a shortened ID number, i.e. into which the third and fourth digits of the original "ship number" field were extracted. Shortened numbers 01-26 have also been converted into a letter (A-Z) (see Fig. 4) because we noted that the first few of these correspond with historical Ocean Weather Station locations (i.e. A, B, C, D, E) (see Diaz et al. 1987). The red over North America corresponds to ship number 0040. ![]() Figure 4. Temporal distribution of the Ocean Station Vessel (OSV) and ice island component of DSI1117, with individual ships plotted separately according to the shortened ID number with 01-26 converted to A-Z plotted in Fig. 3. For reasons discussed in the Fig. 3 caption, 01-27 have been assigned PT=3 (ocean station vessel--on station), and the remaining numbers 40, 48, 49, and 50 PT=9 (ice station). Ice island ship numbers were 40, 0040, 4848, 4949, and 5050 while the first two characters of OSV ship numbers varied across and within shortened ID numbers. Additional ID notes: The original ship number field is described in AWS-WB (1964) as: "All ships have been assigned numbers beginning with 'X' in col. 1. The three following numbers in col. 2 thru 4 range from 001-999. Ship name and records available at NWRC."NCDC (2003) provides the same description, except omitting the last sentence. The original card deck 110 format contained a 3-digit ID described in AWS-WB (1958) as coded 0-999, and in addition as an "Arbitrary number assigned to Navy ships. Whenever possible the ship's number was assigned in numerical sequence, according to the alphabetical arrangement of the ship's name."While as noted above deck 117 represents a continuation of deck 110, whether there is any relationship between their (effectively) 3-digit ship number systems (leaving aside the leading "X" in deck 117) is not known. The remainder of this webpage discusses detailed field comparison results (Fig. 5) between AWS-WB (1964) and NCDC (1968) versions of deck 117, and the Annex provides selected further comparisons between the AWS-WB (1964) and Release 1 versions. While the FOSDIC and TD1100 data sources share some field characteristics (when duplicates are compared), other TD1100 fields did not appear to have been derived exclusively, or in a way that can be replicated, from the otherwise matching FOSDIC reports. As one possible cause, it appears that extensive quality control processing may have been applied to the TD1100 data, or possibly both were derived from a third source. As a consequence, for utilization in upcoming ICOADS updates, it may therefore be advisable to use an original card deck version of the data, unless NCDC is able to locate additional clarifications as to the origins of the TD1100 data, or e.g. original US Navy records (forms) can be checked to better resolve these questions. ![]() Figure 5. Element agreement and number non-missing. FOSDIC and TD1100 reports were considered duplicates if year (YR), month (MO), day (DY), hour (HR), latitude (LAT), longitude (LON), call sign (ID), sea level pressure (SLP), air temperature (AT), wet bulb temperature (WBT), and sea surface temperature (SST) agreed. 84.5995% of the FOSDIC and 99.6504% of the TD1100 reports were duplicates of each other. 2. Discussion of field disagreements (Fig. 5)
![]() Figure 6. Number of reports per year in the FOSDIC version of deck 117, plotted separately to show the trimming flag value, within those reports, for these variables: SF = sea surface temperature, AF = air temperature, UF and VF = wind U and V components, PF = sea level pressure, RF = relative humidity. The trimming flags are defined as follows, where a1 is the individual observation under scrutiny, g is the smoothed median, and s1 and s5 are the smoothed lower and upper median deviation: F = data missing or not computable E = data unusable (SF, AF, and PF, only) C = limits missing (ocean/coastal box) 7 = greater than 4.5 sigma upper limit (a1 > g + 4.5*s5) 6 = less than 4.5 sigma lower limit (a1 < g - 4.5*s1) 5 = greater than 3.5 sigma upper limit (g + 3.5*s5 < a1 <= g + 4.5*s5) 4 = less than 3.5 sigma lower limit (g - 4.5*s1 <= a1 < g - 3.5*s1) 3 = greater than 2.8 sigma upper limit (g + 2.8*s5 < a1 <= g + 3.5*s5) 2 = less than 2.8 sigma lower limit (g - 3.5*s1 <= a1 < g - 2.8*s1) 1 = within 2.8 sigma limits (g - 2.8*s1 <= a1 <= g + 2.8*s5) Annex Additional comparisons between the FOSDIC (SID=141) and Release 1 (SID=9) versions of deck 117. All these examples are from SID=9, OSV Supplement, which during Release 1 processing was converted from TD-1100 format. A W D A A DD L L T N S P B WP D S T TB B D S UU Y M D H A O ITTLDV I I I CD W V V WW L PI AT BT P S S NCH CC W W W S S S T TS 1 B C I P PPTPW R O Y R T N MCIISS D I D 1I DI WI V W1 PA PT TI TI T I TNHLIHMH D P H D P H I LI 0 1 K D T SCCBX 1952 9 9 0 2800 21500 0201 95721 3 684123 97 10221 6 222 189 172 22886 5 7 8 7 165 117141 0 1952 9 9 0 2850 21450 0201 95721 2 656123098 10221 0 222 189 172 22888105AA 7 9 7 165 23585117 9 3 0 1 1952 9 9 300 2800 21500 0201 95721 3 684123 97 10218 6 222 194 178 23985 5 7 8 6 165 117141 0 1952 9 9 300 2850 21450 0201 95721 2 656123098 10218 0 222 194 180 23988505AA 7 9 6 165 23585117 9 3 0 1 1952 9 9 600 2800 21500 0201 95721 3 684118 97 10238 6 222 189 172 24486 5 7 8 6 165 117141 0 1952 9 9 600 2850 21450 0201 95721 2 656118098 10238 0 222 189 172 24488505AA 7 9 6 165 23585117 9 3 0 1 1952 9 9 900 2800 21500 0201 95721 3 684103 9780 10241 6 211 189 178 22888 5 99 165 117141 0 1952 9 9 900 2850 21450 0201 95721 2 65610309880 10241 0 211 189 178 22889A05AA37 0 165 23585117 9 3 0 1 1952 9 91200 2800 21500 0201 95721 3 684 87 97 10227 6 217 189 172 23388 5 99 165 117141 0 1952 9 91200 2850 21450 0201 95721 2 656 87098 10227 0 217 189 176 23389A05AA37 0 165 23585117 9 3 0 1 1952 9 91500 2800 21500 0201 95721 3 684 82 97 10231 6 217 183 161 25062 5 7 8 5 165 117141 0 1952 9 91500 2850 21450 0201 95721 2 656 82098 10231 0 217 183 165 2506650500 7 9 5 165 23585117 9 3 0 1 1952 9 91800 2800 21500 0201 95721 3 684 98 97 10246 6 222 183 161 24461 5 6 6 6 165 117141 0 1952 9 91800 2850 21450 0201 95721 2 656 98098 10246 0 222 183 161 2446650500 6 7 6 165 23585117 9 3 0 1 1952 9 92100 2800 21500 0201 95721 3 684 98 98 10244 6 233 194 172 23311 5 6 8 6 165 117141 0 1952 9 92100 2850 21450 0201 95721 2 656 98098 10244 0 233 194 174 2331150500 6 9 6 165 23585117 9 3 0 1 1952 910 0 2800 21500 0201 95721 3 904 98 98 10233 6 228 194 178 23333 5 8 8 4 165 117141 0 1952 910 0 2850 21450 0201 95721 2 906 98098 10233 0 228 194 178 2333310500 8 9 4 165 23585117 9 3 0 1 1952 910 300 2800 21500 0201 95721 3 454 77 98 10224 6 217 183 167 24422 5 8 8 4 165 117141 0 1952 910 300 2850 21450 0201 95721 2 456 77098 10224 0 217 183 165 2442210500 8 9 4 165 23585117 9 3 0 1 1952 910 600 2800 21500 0201 95721 3 904 82 97 10233 6 217 189 172 24422 5 8 8 4 165 117141 0 1952 910 600 2850 21450 0201 95721 2 906 82098 10233 0 217 189 176 2442210500 8 9 4 165 23585117 9 3 0 1 1952 910 900 2800 21500 0201 95721 3 684 98 97 10236 6 222 194 172 24462 5 8 8 4 165 117141 0 1952 910 900 2850 21450 0201 95721 2 656 98098 10236 0 222 194 180 2446210550 8 9 4 165 23585117 9 3 0 1 1952 9101200 2800 21500 0201 95721 3 684 82 97 10221 6 217 189 172 24421 5 8 8 4 165 117141 0 1952 9101200 2850 21450 0201 95721 2 656 82098 10221 0 217 189 176 2442110550 8 9 4 165 23585117 9 3 0 1 1952 9101500 2800 21500 0201 95721 3 684 36 97 10216 6 217 189 172 24462 5 8 8 4 165 117141 0 1952 9101500 2850 21450 0201 95721 2 656 36098 10216 0 217 189 176 2446650500 8 9 4 165 23585117 9 3 0 1 1952 9101800 2800 21500 0201 95721 3 684 67 98 10227 6 222 200 189 23342 5 6 8 3 165 117141 0 1952 9101800 2850 21450 0201 95721 2 656 67098 10227 0 222 200 190 2334210550 6 9 3 165 23585117 9 3 0 1 1952 9102100 2800 21500 0201 95721 3 904 72 98 10225 6 233 194 178 23343 5 7 8 3 165 117141 0 1952 9102100 2850 21450 0201 95721 2 906 72098 10225 0 233 194 174 2334310550 7 9 3 165 23585117 9 3 0 1 1952 911 0 2800 21500 0201 95721 3 904 77 98 10213 6 228 200 183 23365 5 9 6 3 165 117141 0 1952 911 0 2850 21450 0201 95721 2 906 77098 10213 0 228 200 186 2336510550 9 7 3 165 23585117 9 3 0 1 1952 911 300 2800 21500 0201 95721 3 684 51 98 10203 6 222 194 189 23332 5 8 6 3 165 117141 0 1952 911 300 2850 21450 0201 95721 2 656 51098 10203 0 222 194 180 2333310500 8 7 3 165 23585117 9 3 0 1 1952 911 600 2800 21500 0201 95721 3 904 62 98 10217 6 222 200 189 23322 5 3899 3 165 117141 0 1952 911 600 2850 21450 0201 95721 2 906 62098 10217 0 222 200 190 2332210500 0 3 165 23585117 9 3 0 1 1952 911 900 2800 21500 0201 95721 3 904 46 97 10217 6 222 200 189 23322 5 3899 3 165 117141 0 1952 911 900 2850 21450 0201 95721 2 906 46098 10217 0 222 200 190 2332210500 0 3 165 23585117 9 3 0 1 1952 9111200 2800 21500 0201 95721 3 904 57 97 10213 6 222 200 189 22285 5 3899 3 165 117141 0 1952 9111200 2850 21450 0201 95721 2 906 57098 10213 0 222 200 190 22288505AA 0 3 165 23585117 9 3 0 1 1952 9111500 2800 21500 0201 95721 3 904 51 98 10208 6 222 200 189 23362 5 8 6 2 165 117141 0 1952 9111500 2850 21450 0201 95721 2 906 51098 10208 0 222 200 190 2336650500 8 7 2 165 23585117 9 3 0 1 1952 9111800 2800 21500 0201 95721 3 904 46 98 10223 6 228 189 167 23362 5 8 6 2 165 117141 0 1952 9111800 2850 21450 0201 95721 2 906 46098 10223 0 228 189 169 2336650500 8 7 2 165 23585117 9 3 0 1 1952 9112100 2800 21500 0201 95721 3 904 51 98 10225 6 233 200 183 23372 5 8 8 4 165 117141 0 1952 9112100 2850 21450 0201 95721 2 906 51098 10225 0 233 200 185 2337750500 8 9 4 165 23585117 9 3 0 1References AWS-WB (Air Weather Service and Weather Bureau), 1957: Reference Manual 117: Navy Marine Surface Observations 1951-. HQ, Air Weather Service, Data Control Div., and Weather Bureau, Climatological Services Div. (September 1957), 3 pp. (pdf; 226KB) AWS-WB (Air Weather Service and Weather Bureau), 1958: Reference Manual 110: Navy Marine Surface Obs (WBAN 11). HQ, Air Weather Service, Weather Bureau, Clim. and Hydro. Services (revised 30 June 1958), 3 pp. (pdf; 390KB) AWS-WB (Air Weather Service and Weather Bureau), 1964: Reference Manual 117: Navy Marine Surface Observations 1951-6/63. HQ, Air Weather Service, Data Control Div., and Weather Bureau, Climatological Services Div. (Revised September 1964), 3 pp. (pdf; 260KB) Diaz, H.F., C.S. Ramage, S.D. Woodruff, and T.S. Parker, 1987: Climatic Summaries of Ocean Weather Stations. NOAA Environmental Research Laboratories, Climate Research Division, Boulder, CO, 363 pp. (pdf; 13.2MB) Elms, J.D., S.D. Woodruff, S.J. Worley, and C. Hanson, 1993: Digitizing Historical Records for the Comprehensive Ocean-Atmosphere Data Set (COADS). Earth System Monitor, 4(2), 4-10 (html version) Greenough, M.L.,1998: Report on Conversion of NCDC Microfilms to Electronic Files with FOSDIC. Greenough Data Associates, Rockville, MD. (pdf; 8MB) NCDC (National Climatic Data Center), 1968: TDF-11 Reference Manual. NCDC, Asheville, NC, 138 pp. (pdf; 2.5MB) NCDC (National Climatic Data Center), 2003: Data Documentation for Dataset 1117 (DSI-1117): Navy/Marine Surface Observations (November 17, 2003). (pdf; 42KB) [NCDC version of AWS-WB 1964] Slutz, R.J., S.J. Lubker, J.D. Hiscox, S.D. Woodruff, R.L. Jenne, D.H. Joseph, P.M. Steurer, and J.D. Elms, 1985: Comprehensive Ocean-Atmosphere Data Set; Release 1. NOAA Environmental Research Laboratories, Climate Research Program, Boulder, CO, 268 pp. (NTIS PB86-105723) (html version) [Documentation and Software][Links to additional] U.S. National Oceanic and Atmospheric Administration hosts the icoads website privacy disclaimer Document maintained by icoads@noaa.gov Updated: Jul 25, 2012 18:06:59 UTC http://www.icoads.noaa.gov/deck117.html |