![]()
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ICOADS Web information page (Wednesday, 20-Jul-2016 18:16:38 UTC): Release 3 Data Characteristics1. Quality Control ("trimming") assessmentFor all new historical and contemporary R3.0 input sources, collectively covering the period 1699-2014, Figure 1-1 (note: interactive plot) shows how each deck (DCK) and source ID (SID) responded to the "trimming" quality control processing that is regularly applied to ICOADS (based on three climatological periods, ref. icoads.noaa.gov/dsul.html). The most concerning feature illustrated by these plots is trimming limits missing in remote areas observed by some of the early historical expeditions digitized by ACRE, and in modern GTS buoy data. During translation, we note that only some of the input sources were subject to removal of landlocked reports. ![]() Figure 1-1. Beta4 percentage of trimming flag values per deck and source ID new in Release 3 (bars; left axis) for sea surface temperature (SF), air temperature (AF), u-wind (UF), v-wind (VF), sea level pressure (PF), and relative humidity (RF). Similarly the log10 of the number of non-missing values of the trimming flag (line; right axis). <4.5σ, >4.5σ, limits missing, landlocked, and unusable are trimmed (excluded) from monthly summaries. For comparison, Figures 7a-f at icoads.noaa.gov/r2.5.html present similar information for the R2.5 output data. (Note: this is a dynamic plot: mouse over links to reveal additional patterns.) ![]() ![]() Figure 1-2. Beta4 percentage per deck and source ID new in Release 3 (bars; left axis) of platform type (PT), ID indicator (II), and source exclusion flag (QCZ). Similarly the log10 of the total number of PT, II, or QCZ (line; right axis). The white portion of bars denotes missing. (Note: this is a dynamic plot: mouse over links to reveal additional patterns.) 2. Duplicate Elimination ![]() Figure 2-1. Beta4 percentage of duplicate status (DUPS) values per deck (110-197, 197-715, 720-889, 889-999) and source ID. These are all DUPS entries that appeared where "CERTAIN" and "UNCERTAIN" pertain to the "WEATHER ELEMENT" certainty (number of weather elements in common and number different among W, VV, WW, W1, SLP, AT, and SST): 0 UNIQUE 1 BEST DUPLICATE 2 BEST DUPLICATE WITH SUBSTITUTION 4 UNCERTAIN WEATHER ELEMENT 8 CERTAIN WEATHER ELEMENT 9 UNCERTAIN WEATHER ELEMENT WITH TIME/SPACE 10 CERTAIN WEATHER ELEMENT WITH TIME/SPACE 11 TIME/SPACE/ID 12 UNCERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID 13 CERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID 14 AUTOMATIC REJECTIONDUPS values 0-2 are plotted as positive percentages, because reports with those DUPS values are output into the final user file; whereas, DUPS values >2 are eliminated from the final user file, and thus plotted as negative percentages. (Note: this is a dynamic plot: mouse over links to reveal additional patterns.) ![]() ![]() Figure 2-2. Beta4 percentage (bars; left axis) of platform type (PT) values and number of reports (line; right axis) per year. Similarly for number of platform type values. (Note: this is a dynamic plot: mouse over links to reveal additional patterns.) 3. Map comparisons Figure 3. Beta4 decadal plots (1800-2014) illustrating data additions between Releases 1, 2.5, and 3.0. The colors show the total number of observations in a 2° box per decade separately for four primary variables (and using the enhanced monthly summaries derived using Release 1 trimming limits for both Releases 2.5 and 3.0). Plots for variables with no data in a decade have been omitted (note: Release 1 data began in 1854 and ended in 1979, Release 2.5 ended in 2007, and Release 3.0 ended in 2014). 4. 4-panel plots Figures 4a through 4i provide 4-panel comparison plots between Release 2.5 and Release 3.0 Beta4 (2° enhanced monthly summaries derived using Release 1 trimming limits in both cases). For details on the 4-panel plot layout see icoads.noaa.gov/panels.html.
5. Unique report ID (UID) Table 1. Release 2.5.1 is the original R2.5 with UIDs added (plus a few other changes), and Release 2.5.2 is a blend of the 2008-14 NCEP and NCEI near-real-time GTS data with UIDs added as follows. Note: all numbering refers to the "intermediate" files (or for R3.0.0 to the similar "total" files), which retain all duplicates and other suspect reports; whereas the final files have duplicates, landlocked reports, etc. removed).
Appendix A: Edited Cloud Report (Ecr) attm Differences between sample results produced by the {H99} (henceforth CDIAC) method (http://cdiac.ornl.gov/ftp/ndp026c/ship_199801_200812/DEC08i.Z), versus corresponding output from the original mkECRat.f (routines received from Carole Hahn 8/18/2008, which are being adapted to produce the IMMA Ecr attm), are documented below. Based on these results, some small errors were identified both in the CDIAC method (note: no software corresponding to this method has been located) and mkECRat.f software. In response, the software to produce the Ecr attm has been modified, but we no longer have any independent benchmark data on which to base further automated comparisons. Further information and software will however be provided here at a later date.
![]() ![]() References {H99} Hahn, C.J., and S.G. Warren, 1999: Extended Edited Synoptic Cloud Reports from Ships and Land Stations Over the Globe, 1952-1996. NDP-026C, Carbon Dioxide Information Analysis Center, Oak Ridge National Laboratory, Oak Ridge, TN. (Also available from Data Support Section, National Center for Atmospheric Research, Boulder, CO.) Appendix B: stationmetadata
Appendix C: Release 2.5.2 2008-14 Release 2.5.2 is a blend of the 2008-14 NCEP and NCEI near-real-time GTS data with UIDs added. Since GTS formats CMAN and CREX do not contain location, latitude and longitude were acquired from activestations.xml (2013-14) or stationmetadata.xml edited1 (2008-12). If ID was missing in these files, an NCEP location was used for NCEP data but NCEI data were discarded. ![]() Figure C1. Number of IDs missing per day 2008-12 in activestations2 versus stationmetadata ed. of NCEP GTS data in the CMAN or CREX formats. The 20111020-24 dip does not correspond to a dip in data volume (see Fig. C3). Table C1. IDs most often missing in stationmetadata ed. of NCEP GTS data in the CMAN or CREX formats and NCEI GTS data in the CMAN format. NCEP numbers refer to days missing 2008-12 (1827 days) and NCEI numbers to months in which ID was missing at least once mutiplied by 31 (1798 days due to data problems in 2008/2 and 2008/43, see Fig. C3).
![]() Figure C2. Duplicate elimination intermediate output 2008-2014/8 (bars) by duplicate status (colors) per GTS source and format (horizontal axis): final output above the zero line and worse duplicates eliminated from final output below. Deck does not necessarily reflect platform type (see Fig. C4). deck duplicate status 792 NCEP GTS: Ship Data 0 unique 992 NCEI GTS: Ship Data 1 best duplicate 793 NCEP GTS: Buoy Data (transmitted in FM 13 "SHIP" code) 4 uncertain weather element 993 NCEI GTS: Buoy Data (transmitted in FM 13 "SHIP" code) 8 certain weather element 794 NCEP GTS: Buoy Data (transmitted in FM 18 "BUOY" code) 9 uncertain weather element with time/space 994 NCEI GTS: Buoy Data (transmitted in FM 18 "BUOY" code) 10 certain weather element with time/space 795 NCEP GTS: Coastal-Marine Automated Network (C-MAN code) Data 11 time/space/id 995 NCEI GTS: Coastal-Marine Automated Network (C-MAN code) Data 12 uncertain weather element with time/space/id 797 NCEP GTS: CREX code 13 certain weather element with time/space/id ![]() Figure C3. Platform types per month3 determined by stationmetadata ed. 2008-12 and multiple activestations files 2013-2015/2 of final output. Lightship platform type was identified only by activestations. Gray bars are number of platform type missing and colored lines are number of: yellow 4 - lightship gray 5 - ship blue 6 - moored buoy green 7 - drifting buoy red 13 - Coastal-Marine Automated Network (C-MAN) (NDBC operated) purple 14 - other coastal/island station black 15 - fixed ocean platform (plat, rig) orange 16 - tide gauge ![]() Figure C4. Platform types per deck of final output 2008-2014/8. ![]() Figure C5. August 2014 final output platform types 4 - lightship (c. 50°N 0°E), 5 - ship, 6 - moored buoy, 7 - drifting buoy, 13 - Coastal-Marine Automated Network, 14 - other coastal/island station, 15 - fixed ocean platform, and 16 - tide gauge. ![]() Figure C6. Thousands of platform types per year of final output 2008-2014/8 minus NCEP (except deck 797) input. Duplicates within the NCEP data decrement the bars of additional NCEI data. Negative ship values occur during the Canadian duplicates time period (see merge Examples 1.1). Negative buoy values are explained by NCEP BUOY format data with duplicate statuses of 8: A W D A A D L L T N S P B WP D S T TB B D S U Y M D H A O ITTLDV I I I CD W V V WW L PI AT BT P S S NCH CC W W W S S S T TS 1 B C I P P R O Y R T N MCIISS D I D 1I DI WI V W1 PA PT TI TI T I TNHLIHMH D P H D P H I LI 0 1 K D T S 2011 7151200 -186 26447 1325 332322 01504 82 0 227 1 204 227 165 34815794103 6 8 2011 7151200 -186 26447 1325 332322 01504 82 0 227 1 204 227 165 34815994114 613 2011 7151200 -186 26448 1325 332322 01504 82 0 227 1 204 227 165 34815794103 6 1 2011 7151200 -186 26448 1325 332322 01504 82 0 227 1 204 227 165 34815994114 613 2011 7 71100-5585 22583 1225 371568 101967 40 13 165 52454794103 7 8 2011 7 71100-5585 22583 1225 371568 101967 40 13 165 52454994114 713 2011 7 71100-5587 22586 1225 371568 101967 40 13 165 52454794103 7 1 2011 7 71100-5587 22586 1225 371568 101967 40 13 165 52454994114 713 ![]() Figure C7. ID patterns in NCEP and NCEI input and final output 2008-2014/8. A mask ID is of the form [A-Z][A-Z][A-Z][A-Z][A-Z][0-9][0-9]. 1stationmetadata created="2014-08-18T05:00:02Z" edited to remove 54 stations that did not match activestations2 at 0.1° resolution (i.e., APNM4, CHII2, DPIA1, ESPP4, EVMC1, FPKG1, FPTM4, FRFN7, LOPW1, MAQT2, PBLW1, PBPA2, PNGW3, QPTR1, RSJT2, SANF1, SGNW3, SISA2, TCMW1, THRO1, VAKF1, WEQM1, WEXM1, 21402, 21414, 41037, 41052, 41060, 41NT0, 42099, 42361, 42362, 42369, 42382, 42387, 42877, 42889, 42899, 42921, 42927, 42930, 43412, 45004, 46229, 46246, 46248, 46407, 51000, 51003, 54401, 55012, 55015, 56001, 56003). 2activestations.xml created="2014-08-18T17:20:01UTC" 3 NCEI CMAN format data were missing 2008/4; 2008/2 data had recurring receipt headers of "13 Feb 2008 11:53" including erroneous days 30-31 which resulted in ID being "missing" in stationmetadata. Appendix D: US National Data Buoy Center (NDBC) delayed mode data (deck 883) versus GTS sources (decks 793/993) ![]() Figure D1. Sea surface temperature (air temperature, sea level pressure) per date (the labeling on the horizontal axis represents YYYYMMDDHHHH with hour to hundredths) of Release 2.5.2 GTS decks 793 and 993 (the deck 793 curve falls under the deck 993 curve with both missing after 2012/09/11 5.83 UTC) and delayed mode deck 883 buoy 41004. And similarly for buoy 51000 (sea surface temperature, air temperature, sea level pressure) and C-MAN TPLM2 (sea surface temperature, air temperature, sea level pressure) (the decks 793(5) and 993(5) curves are identical and the deck 883 curve never missing). The 41004 deck 883 lag may be related to inconsistent hour between GTS groups YYGGiw and 9GGgg (see Fig. D3): BBXX01090120120903SNVD01KWBC010800RRN////// 41004 01081 99325 70791 46/// /2603 10268 20232 40205 56007 90050 22200For reference from the WMO Manual on Codes No. 306, relevant regulations regarding these date/time groups: "12.1.6 The actual time of observation [i.e. GG from YYGGiw] shall be the time at which the barometer is read. [...] 12.2.8 Group 9GGgg This group shall be included: (a) When the actual time of observation differs by more than 10 minutes from the standard time GG reported in Section 0; (b) When additionally specified by regional decision. Note: See Regulation 12.1.6." ![]() Figure D2. A zoom in on the end of Fig. D1 showing a possible moving threshold for SST identified by Liz Kent of the UK National Oceanography Centre (see also Fig. D1, 51000, SST). ![]() Figure D3. 41004 GTS hour to hundredths minus GTS whole hour per month in 2012. 41004 was the only ID in 2012/09 of Release 2.5.2 intermediate output with a pattern of inconsistent hour (see Fig. D1) and that pattern in 41004 was limited to 2012. Expected differences are -0.17 and 23.83. Appendix E: Global Tropical Moored Buoy Array (GTMBA) Table E1. Release 2.5 Pacific Marine Environmental Laboratory (PMEL) decks and number of reports.
![]() Figure E1. Release 2.5 wind direction (from; meteorological convention) within 1° of 0°N 140°W (easterly trade winds) by deck. Erroneous wind direction (to; oceanographic convention) in deck 145 was discovered by Liz Kent of the UK National Oceanography Centre. Appendix F: Climatological Database for the World's Oceans (CLIWOC) Release 2.5 deck 730 (not subject to duplicate elimination) IDs are right-justified: A W D A A L L T N S P B WP D S T TB B D Y M D H A O ITTLDV I I I CD W V V WW L PI AT BT P S S NCH CC W W W S S S T TS 1 B C R O Y R T N MCIISS D I D 1I DI WI V W1 PA PT TI TI T I TNHLIHMH D P H D P H I LI 0 1 K 1793 8 1 300 -21 13835 1306 10 1085FR 935 67 165 335 8730 1793 8 11200 7530 135 1306 10 825NL 84 165 7051730 1793 8 11400 3780 32438 1306 10 1545UK 262 7 250 165 21075730 1793 8 11400 3777 32464 1306 10 1858NL 206 165 21075730 1793 8 11400 1220 33502 1306 10 1117UK 455 26 7 258 165 28324730 1793 8 11400-1577 32753 1306 10 546ES 1105 93 165 39052730
[Data and Documentation][Documentation and Software]
U.S. National Oceanic and Atmospheric Administration hosts the icoads website privacy disclaimer Document maintained by icoads@noaa.gov Updated: Jul 20, 2016 18:16:38 UTC http://www.icoads.noaa.gov/r3.html |