ICOADS Web information page (Wednesday, 11-May-2016 19:29:24 UTC):

Release 2.5 (Preliminary) wind speed (W) data continuity questions

Figs. 1a-1c provide 4-panel comparison plots between preliminary Release 2.5 data and Release 2.4 data (2° enhanced monthly summaries in both cases) for three selected 20-year focus periods (for details on the 4-panel plot layout see icoads.noaa.gov/panels.html; note that a minimum of 1000 nobs are required per month, or all curves are omitted to ensure meaningful data patterns). The remaining plots explore some of the noteworthy data-difference patterns on a deck-by-deck basis for selected periods or subperiods.

Discussion
Note: "main" figure references are to the parent [Release 2.5 (Preliminary) Data Characteristics].

a. Fig. 1a shows positive wind speed differences around 1883-87, probably associated with deck 704 ("US Marine Met. Journals"), which provides a large influx of new data. The differences may be also associated with new deck 720 ("Deutscher Wetterdienst (DWD) Marine Meteorological Archive"; see main Fig. 4b). During 1880-89 Fig. 2a indicates some suspicious distributions in other decks, e.g., decks 732 ("Russian MORMET"), 762 ("Japanese Kobe Collection", and 901 ("FOSDIC Reconstructions"), which might for example have arisen from early units translation (e.g., Beaufort force code) or archival issues.

Proposed action:
None at this time.

b. Fig. 1b shows strongly negative wind speed differences around World War II. Fig. 2b (see also main Fig. 4b) suggest that these differences arise from decks 780 (WOD05) and 245 (recently keyed UK "Royal Navy Ship's Logs 1938-47"). Also from Fig. 2b, some other preexisting ICOADS decks at reduced frequency such as deck 197 ("Danish (and Other) Marine (Polar)") and UK deck 206 ("Ocean Weather Stations (OWS) (1930 code)") could be worthy of further investigation to check for possible data inhomogeneities (such as early units-translation issues).

Proposed actions:
(i) For deck 780, MBT data will be reprocessed to correct or eliminate many erroneous zero wind speed values (apparently related to WOD05 processing errors that combined wind speeds in different units together into one field).
(ii) For deck 245, suggested possible UK work initially to look at the reality of the wind speed distribution illustrated in Fig. 2b (note: a small Beaufort force conversion difference with respect to the regular ICOADS translation scale is known to exist, but believed not to be a significant factor). Potentially as a drastic step all wind speeds could be deleted from deck 245 for Release 2.5 if near-term resolution is unrealistic.

c. Fig. 1c also shows some fairly consistently negative wind speed differences around 1950-69. Similarly to item b, Fig. 2c suggests that these differences originate in deck 780 (WOD05). Separate results (not shown) of deck 780 winds also indicate erroneous zero wind speeds extending beyond 1969 (at least until 1973). Also similarly to item b, some preexisting ICOADS decks such as 761 ("Japanese Whaling Ship Data") and 898 ("Japanese") appear to merit further validation as time permits.

Proposed action:
(See action (i) under b.)

d. Doubled wind speeds.
In 1968 (see Fig. 3 and the Annex) some instances of doubled wind speeds were found by comparing duplicate reports between UK MDB deck 254 ("Int. Maritime Met. (IMM) Data (foreign or unknown origin)") with a US version of these keyed logbook data in deck 128 ("International Marine (US- or foreign-keyed ship data)"). Similarly, in 1960 additional examples (not shown) of a reverse situation (deck 128 wind speeds doubled compared to deck 254) were found (associated with data recruited by Hong Kong). Unfortunately, it isn't obvious in either case which copy of the data was translated correctly (i.e., from reported whole m/s or whole knots), and probably most of these early international exchange format (IMMPC or IMMT) data no longer exist anywhere. Moreover these appear to be symptoms of more widespread wind speed differences in early IMM data, since for instance we also have evidence of generally poor wind speed agreement between these two decks throughout 1960. While dupelim processing presently offers only limited capabilities to assess in detail how data from these and other IMM or related decks (e.g., US deck 926 and Russian MORMET deck 732) match up, development in the future of additional such capabilities (e.g., possibly providing measures of how well one deck compares against multiple other decks) might be very helpful for improving data quality.

Proposed action:
As discussed on the SST webpage, dupelim rules will be adjusted to preferentially select most other decks over deck 254 in the event of near-equivalent matches.

e. Impacts arising from different Beaufort force codes and equivalency scales.
Figures 2a-2e suggest that some additional historical decks (as also discussed under items a-c above) might contain unresolved wind speed inhomogeneities. In some cases unusual data distributions in earlier estimated wind data can arise from known differences in original wind force reporting procedures or codes. For example, deck 702 ("Norwegian Logbook Collection"; see Fig. 2a) was originally reported in a 0-6 scale (with the force numbers representing one half of the Beaufort scale numbers). Similarly, some deck 197 ("Danish (and Other) Marine (Polar)"; see Fig. 2b) data were originally reported in the "Ben Nevis Observatory Code." Beyond the original reporting procedures, an important contributing issue may be the uniformity of the equivalency scales used for translation of wind force codes (or terms for pre-Beaufort data) into modern scientific units (i.e., ultimately into m/s in IMMA format). The largest known such inhomogeneity in ICOADS data should already have been resolved during earlier COADS Release 1 (and some follow-up ICOADS Release 2.0) processing, which rebracketed HSST data from the Indian (Netherlands responsibility) and Atlantic (German responsibility) basins (decks 155-156) back into the "Old" (WMO Code 1100) Beaufort scale from the "New" (so-called "Scientific" scale recommended by WMO/CMM-IV). Following COADS Release 1, however, newly available historical data sources (e.g., some decks from the UK Marine Data Bank or other international sources) are likely adding to this data mixture issue through additional equivalence scale variations applied to the data prior to blending into ICOADS.

Proposed action:
None at this time, however a longstanding recommendation from ICOADS workshops has been to continue to seek improved adjustments (to both measured and estimated wind speeds), including possibly the adoption of a modernized equivalency scale to replace WMO Code 1100 as appropriate in ICOADS. A useful step envisioned toward this latter proposed action could be finalization of an IMMA historical attachment to include originally reported Beaufort force codes (recovered as feasible from surviving supplementary data).

Figure 1a

Figure 1b

Figure 1c

Figure 2a. Bars show percentage frequency (left axis) of wind speed values per deck (all ICOADS deck numbers are listed in Tables D6a-D6c). The line (right axis) indicates the total number of observations per deck during the period (note: log scale). Decks are ranked left to right from the lowest frequency to highest. Deck believed to contribute most heavily to anomaly changes: Deck 704 (US Marine Meteorological Journals Collection (1878-94)). Since total wind speed values (not anomalies) are plotted here, anomalous patterns could result purely from geographic effects.

Figure 2b. (See Figure 2a for plot explanation.) Decks believed to contribute most heavily to anomaly changes: 780 (NODC/OCL World Ocean Database (WOD) (and formerly Atlas, WOA)) and 245 (Royal Navy Ship's Logs 1938-47).

Figure 2c. (See Figure 2a for plot explanation.) Deck believed to contribute most heavily to anomaly changes: 780 (NODC/OCL World Ocean Database (WOD) (and formerly Atlas, WOA)).

Figure 2d. (See Figure 2a for plot explanation; this figure presents deck patterns for the entire currently processed period of record.)

Figure 2e. Bars show percentage frequency (left axis) of wind speed values per year. The line (right axis) indicates the total number of observations per year (note: log scale). Since total wind speed values (not anomalies) are plotted here, anomalous patterns could result purely from geographic effects.

Figure 3. Wind speed quotients for the indicated matches of duplicate reports between decks 254 and 128 (April 1968), illustrating the doubled wind speed relationship between the two decks.

Annex: Duplicate report match examples (note: field abbreviations, listed vertically, are defined here) between decks:

254: Int. Maritime Met. (IMM) Data (foreign or unknown origin)
128: International Marine (US- or foreign-keyed ship data)

illustrating (apparently) doubled wind speeds (W, m/s) in deck 254 (examples with country code C1=25, USSR). These are contrasted with the first and last match examples (with country code C1=17, Japan) where wind speeds basically agree. For the doubled cases, the wind speed indicator (WI, last column) is WI=0 (meter per second, estimated) in deck 254 and WI=6 (estimated (original units unknown)/unknown method) in deck 128. According to national practices we expect that USSR data were originally reported in whole m/s (note: knots = m/s * 1.94384449).

 MATCH= 9 CERTAIN: UNCERTAIN WEATHER ELEMENT WITH TIME/SPACE    COMMON=7 DIFFER=1                                                
                                                                                                                              DD 
                L     LIIIIIIIII                S   P       W   D   S                   S S S    IIIII  R           B  D  S   UU 
   Y M D   H    A     ODDDDDDDDD       V WWW    L   P   A   B   P   S NC CC W W W S S S D P HI ERCCCCC  RTDV C COO  1  C  I P PPW
   R O Y   R    T     N123456789  D  W V W12    PA  P   T   T   T   TNHLHMH D P H D P H 2 2 2S SS12345  RRSS 1 2SP  0  K  D T SCI
1968 4 1   0 5060 20000         13019696 32* 9950****  70  50  27  408872**** 8 815 9 8*********************17***1125254 90 5 114
1968 4 1   0 5060 20000JINJ     13019594 32* 9950****  70  50  27  408872**** 9 815 9 8*********************17****125128  1 0 917
 MATCH= 9 CERTAIN: UNCERTAIN WEATHER ELEMENT WITH TIME/SPACE    COMMON=5 DIFFER=1                                                
                                                                                                                              DD 
                L     LIIIIIIIII                S   P       W   D   S                   S S S    IIIII  R           B  D  S   UU 
   Y M D   H    A     ODDDDDDDDD       V WWW    L   P   A   B   P   S NC CC W W W S S S D P HI ERCCCCC  RTDV C COO  1  C  I P PPW
   R O Y   R    T     N123456789  D  W V W12    PA  P   T   T   T   TNHLHMH D P H D P H 2 2 2S SS12345  RRSS 1 2SP  0  K  D T SCI
1968 4 1   0 3970 14230         19019098 11*10131****************  92511510** 5 417 6 5*********************25**11192254 90 5 120
1968 4 1   0 3970 14230         190 9898 11*10131********************511510** 5 417 6 5*********************25****192128  5 0 926
 MATCH= 9 CERTAIN: UNCERTAIN WEATHER ELEMENT WITH TIME/SPACE    COMMON=5 DIFFER=1                                                
                                                                                                                              DD 
                L     LIIIIIIIII                S   P       W   D   S                   S S S    IIIII  R           B  D  S   UU 
   Y M D   H    A     ODDDDDDDDD       V WWW    L   P   A   B   P   S NC CC W W W S S S D P HI ERCCCCC  RTDV C COO  1  C  I P PPW
   R O Y   R    T     N123456789  D  W V W12    PA  P   T   T   T   TNHLHMH D P H D P H 2 2 2S SS12345  RRSS 1 2SP  0  K  D T SCI
1968 4 1   0 3750 13460         24019098 20*10137****************  22000*00** 4 3***************************25**11191254 90 5 120
1968 4 1   0 3750 13460         240 9898 20*10137********************000900** 5 3 0** 0*********************25****191128  5 0 926
 MATCH= 9 CERTAIN: UNCERTAIN WEATHER ELEMENT WITH TIME/SPACE    COMMON=7 DIFFER=1                                                
                                                                                                                              DD 
                L     LIIIIIIIII                S   P       W   D   S                   S S S    IIIII  R           B  D  S   UU 
   Y M D   H    A     ODDDDDDDDD       V WWW    L   P   A   B   P   S NC CC W W W S S S D P HI ERCCCCC  RTDV C COO  1  C  I P PPW
   R O Y   R    T     N123456789  D  W V W12    PA  P   T   T   T   TNHLHMH D P H D P H 2 2 2S SS12345  RRSS 1 2SP  0  K  D T SCI
1968 4 1   0 3230 17270          60 9899 10*10323**** 137******** 169445600** 5 2***************************25**11195254 90 5 920
1968 4 1   0 3230 17270          60 5199 10*10323**** 137******** 169445600** 5 23410 0*********************25****195128  5 0 126
 MATCH= 9 CERTAIN: UNCERTAIN WEATHER ELEMENT WITH TIME/SPACE    COMMON=7 DIFFER=1                                                
                                                                                                                              DD 
                L     LIIIIIIIII                S   P       W   D   S                   S S S    IIIII  R           B  D  S   UU 
   Y M D   H    A     ODDDDDDDDD       V WWW    L   P   A   B   P   S NC CC W W W S S S D P HI ERCCCCC  RTDV C COO  1  C  I P PPW
   R O Y   R    T     N123456789  D  W V W12    PA  P   T   T   T   TNHLHMH D P H D P H 2 2 2S SS12345  RRSS 1 2SP  0  K  D T SCI
1968 4 1   0 3130 17380          3013998 21*10311**** 174******** 182440950******35 7 5*********************25**11195254 90 5 120
1968 4 1   0 3130 17380          30 7298 21*10311**** 174******** 182440950******35 7 5*********************25****195128  5 0 926
 MATCH= 9 CERTAIN: UNCERTAIN WEATHER ELEMENT WITH TIME/SPACE    COMMON=7 DIFFER=1                                                
                                                                                                                              DD 
                L     LIIIIIIIII                S   P       W   D   S                   S S S    IIIII  R           B  D  S   UU 
   Y M D   H    A     ODDDDDDDDD       V WWW    L   P   A   B   P   S NC CC W W W S S S D P HI ERCCCCC  RTDV C COO  1  C  I P PPW
   R O Y   R    T     N123456789  D  W V W12    PA  P   T   T   T   TNHLHMH D P H D P H 2 2 2S SS12345  RRSS 1 2SP  0  K  D T SCI
1968 4 1   0 2770 17300          6019098 31*10392**** 228******** 195769410** 3 3***************************25**11231254 90 5 920
1968 4 1   0 2770 17300          60 9898 31*10392**** 228******** 195769410** 5 3 0** 0*********************25****231128  5 0 126
 MATCH= 9 CERTAIN: UNCERTAIN WEATHER ELEMENT WITH TIME/SPACE    COMMON=7 DIFFER=1                                                
                                                                                                                              DD 
                L     LIIIIIIIII                S   P       W   D   S                   S S S    IIIII  R           B  D  S   UU 
   Y M D   H    A     ODDDDDDDDD       V WWW    L   P   A   B   P   S NC CC W W W S S S D P HI ERCCCCC  RTDV C COO  1  C  I P PPW
   R O Y   R    T     N123456789  D  W V W12    PA  P   T   T   T   TNHLHMH D P H D P H 2 2 2S SS12345  RRSS 1 2SP  0  K  D T SCI
1968 4 1   0 2510 17320          5019097 21*10250**** 215******** 211551500** 6 4***************************25****231254 90 5 920
1968 4 1   0 2510 17320          50 9897 21*10250**** 215******** 211551500** 5 2 5 6 4*********************25****231128  5 0 126
 MATCH= 9 CERTAIN: UNCERTAIN WEATHER ELEMENT WITH TIME/SPACE    COMMON=6 DIFFER=1                                                
                                                                                                                              DD 
                L     LIIIIIIIII                S   P       W   D   S                   S S S    IIIII  R           B  D  S   UU 
   Y M D   H    A     ODDDDDDDDD       V WWW    L   P   A   B   P   S NC CC W W W S S S D P HI ERCCCCC  RTDV C COO  1  C  I P PPW
   R O Y   R    T     N123456789  D  W V W12    PA  P   T   T   T   TNHLHMH D P H D P H 2 2 2S SS12345  RRSS 1 2SP  0  K  D T SCI
1968 4 1   0 2500 17350          1017097 21********** 210******** 190000*00** 4 3 8 5 4*********************25****231254 90 5 920
1968 4 1   0 2500 17350          10 8797 21********** 210******** 190000900** 5 3 8 5 4*********************25****231128  5 0 126
 MATCH= 9 CERTAIN: UNCERTAIN WEATHER ELEMENT WITH TIME/SPACE    COMMON=7 DIFFER=1                                                
                                                                                                                              DD 
                L     LIIIIIIIII                S   P       W   D   S                   S S S    IIIII  R           B  D  S   UU 
   Y M D   H    A     ODDDDDDDDD       V WWW    L   P   A   B   P   S NC CC W W W S S S D P HI ERCCCCC  RTDV C COO  1  C  I P PPW
   R O Y   R    T     N123456789  D  W V W12    PA  P   T   T   T   TNHLHMH D P H D P H 2 2 2S SS12345  RRSS 1 2SP  0  K  D T SCI
1968 4 1   0-2060  4220         150 8798 20*10110**** 248 225 213 260000*00*********************************21**11398254 90 5 923
1968 4 1   0-2060  4220         150 8298 20*10110**** 248 225 216 260000900*********************************21****398128  1 0 126

[Documentation and Software][Release 2.5 (Preliminary) Data Characteristics]

U.S. National Oceanic and Atmospheric Administration hosts the icoads website privacy disclaimer
Document maintained by icoads@noaa.gov
Updated: May 11, 2016 19:29:24 UTC
http://www.icoads.noaa.gov/pre2.5w.html