| Data and Documentation | Chronology and News | Program Status | Publications | Related Data and Resources | Contact Points |
ICOADS Web information page (Wednesday, 27-Mar-2013 22:51:54 UTC):
Characteristics of a Merged (NCEP-NCDC) Global Telecommunication System (GTS) Near-Real-Time (NRT) Product Duplicate Elimination (dupelim)
Below, detailed results from five different dupelim algorithms are compared: algorithm 1, not requiring IDs to match (see Fig. 1 and Examples 1); algorithm 2, requiring IDs to match (Fig. 2 and Examples 2); algorithm 3, inclusion of deck 797 (Fig. 3 and Examples 3); algorithm 4, deck 797 two tenths difference allowance for wind speed and SST (Fig. 4); and algorithm 5, deck 797 location allowance (Fig. 5). Generally during dupelim processing, all reports are compared with all other reports within a circular buffer, such that a duplicate status (DUPS) flag within each report may be reset (to a higher, but not lower, value) as a result of subsequent matches. Matches therefore refers to all "transient" matches that were made and DUPS results that may be altered later, during this processing. The initial output from dupelim is the "intermediate" file, containing all duplicates (classified either as "certain" or "uncertain"), as flagged by DUPS. From that intermediate output, the final "merged" output is created, generally by removing all but DUPS=0 (unique) or 1 (best duplicate).
Figure 1. Dupelim skill, in combining NCDC GTS and NCEP GTS data, for March 2012 (Algorithm 1). In this test IDs were not required to match.
- Input (NCDC and NCEP) and output from dupelim ("merge"). NCEP deck 794 had 131 more call signs (mostly in the North Atlantic) and generally more reports per call sign, than NCDC deck 994.
- Deck matches. NCDC deck 995 unexpectedly matched itself (see Examples 1.2). Smaller unlabeled blue bars are other decks matching themselves or other platforms. Based on comparison of the supplemental data (SUPD, i.e. original input GTS report(s) used to construct each NCEP BUFR report), and in spite of NCEP's dup-merge processing, 678 NCEP reports were exact duplicates (no triplicates), all BBXX. In contrast no "legitimate" (i.e. well formed) NCDC SUPD were exact duplicates.
- Duplicate status (DUPS) (colors) for the various deck-to-deck matches (horizontal axis) between NCDC and NCEP data, such that these are the DUPS entries that appeared in non-trivial numbers:
8 CERTAIN WEATHER ELEMENT 10 CERTAIN WEATHER ELEMENT WITH TIME/SPACE 13 CERTAIN WEATHER ELEMENT WITH TIME/SPACE/IDDeck 792-992 matches are not DUPS 13 because of NCEP's ship call sign masking. - The horizontal axis lists the number of weather elements (W, VV, WW, W1, SLP, AT, SST) in common (zero different), with the bars stratified by DUPS (colors). Zero weather elements in common with DUPS 8 are extremely weak matches (see Examples 1.2).
Examples 1
These are examples to accompany Fig. 1, listing either (1) input supplementary data (SUPD), or (2) IMMA reports processed through dupelim.
- NCDC SUPD with odd BBXX all without the 99LaLaLa (latitude) group including questionable delayed March receipts received in April (note: ~10K/0.3% of the 3.4M NCDC reports were deleted during dupelim due to missing YR, MO, LON, and/or LAT).
99 0 BBXX01030220121218 X25MSG 40dd00.0d 40dd 311020307984 31102027935611 11 0001 000184 20120302120422 USA COMSAT EIK 99 0 BBXX01031020121530 X25MSG 40dd00.0d 40dd 311020307984 31102027935611 11 0001 000169 20120310152301 USA COMSAT EIK 99 0 BBXX01031520121127 X25MSG 40dd00.0d 40dd 311020307984 31102027935611 11 0001 000580 20120315111615 USA COMSAT EIK 99 0 BBXX01031920120151 X25MSG 40dd00.0d 40dd 311020307984 31102027935611 11 0001 000179 20120319014936 USA COMSAT EIK 99 0 BBXX01032720121034 X25MSG 40dd00.0d 40dd 311020307984 31102027935611 11 0001 001293 20120327102955 USA COMSAT EIK 99 0 BBXX01033020121346 X25MSG 40dd00.0d 40dd 311020307984 31102027935611 11 0001 000583 20120330134359 USA COMSAT EIK 99 0 BBXX01032920121156 0000078001SMVN20 SVMR 290000 AAXX 29001 80402 NIL 99 0 BBXX01033120120051 0000083501SMVN20 SVMR 310000 AAXX 31001 80402 NIL 99 0 BBXX01031420121233 0000123101SMVN20 SVMR 141200 CCA AAXX 14121 80402 NIL 99 0 BBXX01030520120756 0000060901SAVN20 SVMR 050600 METAR SVCR 050600Z NIL 99 0 BBXX01031320120902 0000067501SAVN20 SVMR 130600 METAR SVCR 130600Z NIL 99 0 BBXX01030720120032 0000060601SAVN21 SVMR 070000 METAR SVJC 070000Z NIL 99 0 BBXX01030820120851 0000074701SAVN21 SVMR 080600 METAR SVJC 080600Z NIL 99 0 BBXX01030520120034 0000061801SAVN21 SVMR 050000 METAR SVLO 050000Z NIL 99 0 BBXX01030820120041 0000059201SAVN21 SVMR 080000 METAR SVLO 080000Z NIL 99 0 BBXX01032620120049 0000041301SAVN20 SVMR 260000 SVCR 260000Z NIL 99 0 BBXX01033020120637 80479 NIL 99 0 BBXX01033120120036 80479 NIL 99 0 BBXX01033120121234 AAXX 31121 80400 NIL 99 0 BBXX01033120121823 AAXX 31181 80400 NIL 99 0 BBXX01031320120902 METAR SVAC 130600Z NIL 99 0 BBXX01031720120034 METAR SVAC 170000Z NIL 99 0 BBXX01031420121233 METAR SVBC 141200Z NIL 99 0 BBXX01031820121824 METAR SVBC 181800Z NIL 99 0 BBXX01030920121846 METAR SVCZ 091800Z NIL 99 0 BBXX01032120121840 SAVN20 SVMR 211800 AAXX 21181 80402 NIL 99 0 BBXX01030920121846 SAVN21 SVMR 091800 METAR SVLO 091800Z NIL 99 0 BBXX01032120121840 SAVN21 SVMR 211800 METAR SVJC 211800Z NIL 99 0 BBXX01032620120049 SVAC 260000Z NIL 99 0 BBXX01032620120049 SVBI 260000Z NIL 99 0 BBXX01032620120049 SVBS 260000Z NIL 99 0 BBXX01032620120049 SVCB 260000Z NIL 99 0 BBXX01032620120049 SVCU 260000Z NIL 99 0 BBXX01032620120049 SVFM 260000Z NIL 99 0 BBXX01032620120049 SVMC 260000Z NIL 99 0 BBXX01032620120049 SVMT 260000Z NIL 99 0 BBXX01032620120049 SVSO 260000Z NIL 99 0 BBXX01032620120049 SVVA 260000Z NIL 99 0 BBXX01030220121218 ---BAD 06165a41.IN 99 0 BBXX01031020121530 ---BAD 061ee841.IN 99 0 BBXX01031920120151 ---BAD 0627c741.IN 99 0 BBXX01033020121803 ---BAD 06302341.IN 99 0 BBXX01040320121216 48916 31598 73601 10272 20240 30072 40119 53008 70391 82908 333 00/// 10328 58009 70000 82995 8729 9 99 0 BBXX01040320121231 48826 31595 83402 10227 20218 30015 40150 53009 71022 888// 333 00023 10250 58033 70000 83895 8869 7 99 0 BBXX01040320121247 80410 31356 81104 10210 20192 39435 40110 71022 887// 333 20201 30/// 50594 55074 2//// 562// 58001 88710 555 10147 99 0 BBXX01040320121513 48826 31559 83602 10226 20220 30030 40166 51015 71022 888// 333 58020 83895 88697 99 0 BBXX01040320121836 48839 31692 83601 10210 20207 30086 40152 58013 74444 885// 222// 00194 2//// 333 58007 88696 99 0 BBXX01040320121836 80448 31468 70000 10337 20234 3//// 4//// 71729 870// 333 30/// 569// 81917 86617 99 0 BBXX01040320122131 48808 31496 80000 10202 20199 39855 40138 57018 71022 888// 333 59017 83894 88696 99 0 BBXX01040320122131 48826 31595 82701 10220 20217 30000 40135 56018 71022 888// 333 83895 88697 99 0 BBXX01040420120045 48808 31440 80000 10208 20205 39868 40151 52013 71022 888// 333 00021 20200 3/019 59027 83894 88696 99 0 BBXX01040420120045 48839 31692 83602 10207 20206 30082 40148 53012 74444 885// 222// 00197 2//00 333 01/// 20193 59022 88696 96122 99 0 BBXX01040420120030 80437 31460 70001 10270 20229 39975 40060 72929 875// 333 10347 30/// 569// 59005 87613 555 10066 99 0 BBXX01040420120030 80448 31460 70000 10273 20233 3//// 4//// 70292 8352/ 333 10350 30/// 5699/ 83617 84457 99 0 BBXX01040420120325 48826 31595 70000 10247 20231 30021 40155 51011 71022 8211/ 333 59025 82898 87499 99 0 BBXX01040420120325 48830 31440 81802 10242 20220 39850 40153 50007 71022 888// 333 59029 83894 88697
- NCDC deck 995 matching internally. DUPS/MATCH = 8 as shown below are mostly not genuine duplicates (e.g. PTAW1 and FRDW1 are nearby water level stations). As shown below, most DUPS/MATCH = 13 matches (note: hereafter referred to as "complement duplicates") appear to provide complementary data elements, which likely would not be satisfactorily resolved by the currently planned dupelim processing (unless fully replaced by dup-merged NCEP).
MATCH= 8 CERTAIN: CERTAIN WEATHER ELEMENT WITH NO CROSS COMMON=0 DIFFER=0
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 4855 23699FRDW1 ************100323 30************ 71***A*********************************** ****16599511413 1225 5***0** 1*****64
2012 3 1 0 4813 23656PTAW1 310 77**********************************A*********************************** ****16599511413 8225 504***********64
MATCH= 8 CERTAIN: CERTAIN WEATHER ELEMENT WITH NO CROSS COMMON=0 DIFFER=0
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 4728 23758TCMW1 190 77**********************************A*********************************** ****16599511413 8225 504***********64
2012 3 1 0 4727 23759TCNW1 ************100642 20************ 64***A*********************************** ****16599511413 1225 5***0** 1*****64
MATCH= 8 CERTAIN: CERTAIN WEATHER ELEMENT WITH NO CROSS COMMON=0 DIFFER=0
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 4470 28450OBGN6 ************101436 18 -30***************A*********************************** ****17099511413 1225 5***0*********64
2012 3 1 0 4433 28407ALXN6 ********************************* 14***A*********************************** ****17099511413 8225 5***0** 1*****64
MATCH= 8 CERTAIN: CERTAIN WEATHER ELEMENT WITH NO CROSS COMMON=0 DIFFER=0
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 4070 28598BATN6 ************101276 16 55******** 61***A*********************************** ****17099511413 1225 5***0** 1*****64
2012 3 1 0 4008 28513BDRN4 50 26**********************************A*********************************** ****17099511413 8225 504***********64
MATCH= 8 CERTAIN: CERTAIN WEATHER ELEMENT WITH NO CROSS COMMON=0 DIFFER=0
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 3898 28352APAM2 ************100968 8 87***************A*********************************** ****20699511413 1225 5***0*********64
2012 3 1 0 3832 28355SLIM2 ********************************* 75***A*********************************** ****20699511413 8225 5***0** 1*****64
MATCH= 8 CERTAIN: CERTAIN WEATHER ELEMENT WITH NO CROSS COMMON=0 DIFFER=0
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 3898 28352APAM2 ************100968 8 87***************A*********************************** ****20699511413 1225 5***0*********64
2012 3 1 0 3813 28347PPTM2 290 31**********************************A*********************************** ****20699511413 8225 504***********64
MATCH= 8 CERTAIN: CERTAIN WEATHER ELEMENT WITH NO CROSS COMMON=0 DIFFER=0
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 3878 28329BSLM2 170 15******100908 20 95**** 92*******A*********************************** ****20699511413 1225 504*0*********64
2012 3 1 0 3832 28355SLIM2 ********************************* 75***A*********************************** ****20699511413 8225 5***0** 1*****64
MATCH=11 CERTAIN: TIME/SPACE/ID COMMON=3 DIFFER=2
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 4066 28593ROBN4 60 72******10100**** 52***************A*********************************** ****1709951141311225 504*0*********64
2012 3 1 0 4066 28593ROBN4 70 82******101196 14 52***************A*********************************** ****17099511413 1225 504*0*********64
MATCH=13 CERTAIN: CERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID COMMON=0 DIFFER=0
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 4855 23699FRDW1 ************100323 30************ 71***A*********************************** ****16599511413 1225 5***0** 1*****64
2012 3 1 0 4855 23699FRDW1 280 10*************** 38***************A*********************************** ****1659951141313225 504*0*********64
MATCH=13 CERTAIN: CERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID COMMON=0 DIFFER=0
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 4813 23656PTAW1 ************100291 28 61******** 71***A*********************************** ****16599511413 1225 5***0** 1*****64
2012 3 1 0 4813 23656PTAW1 310 77**********************************A*********************************** ****1659951141313225 504***********64
MATCH=13 CERTAIN: CERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID COMMON=0 DIFFER=0
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 4791 23536LAPW1 ************100361 18 47******** 52***A*********************************** ****16599511413 1225 5***0** 1*****64
2012 3 1 0 4791 23536LAPW1 200 21**********************************A*********************************** ****1659951141313225 504***********64
MATCH=13 CERTAIN: CERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID COMMON=0 DIFFER=0
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 4335 23568CHAO3 ************101501 14 64******** 81***A*********************************** ****16599511413 1225 5***0** 1*****64
2012 3 1 0 4335 23568CHAO3 240 26**********************************A*********************************** ****1659951141313225 504***********64
MATCH=13 CERTAIN: CERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID COMMON=0 DIFFER=0
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 4077 23578HBYC1 ************102113 7************ 84***A*********************************** ****16599511413 1225 5***0** 1*****64
2012 3 1 0 4077 23578HBYC1 220 51*************** 63***************A*********************************** ****1659951141313225 504*0*********64
MATCH=13 CERTAIN: CERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID COMMON=0 DIFFER=0
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 4008 28513BDRN4 ************101126 8 65******** 61***A*********************************** ****17099511413 1225 5***0** 1*****64
2012 3 1 0 4008 28513BDRN4 50 26**********************************A*********************************** ****1709951141313225 504***********64
MATCH=13 CERTAIN: CERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID COMMON=1 DIFFER=0
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 4556 23609TLBO3 270 57**********************************A*********************************** ****1659951141313225 504***********64
2012 3 1 0 4556 23609TLBO3 270 57******10094**** 45******** 73***A*********************************** ****16599511413 1225 504*0** 1*****64
Figure 2. Dupelim skill, in combining NCDC GTS and NCEP GTS data, for March 2012 (Algorithm 2). In this test, IDs were required to match for two reports to be considered duplicates, except in the case of 792-992 matches (which cannot be required to match, due to NCEP's ship call sign masking).
- Input (NCDC and NCEP) and output from dupelim ("merge").
- Deck composition (colors) of the intermediate output from dupelim for each DUPS (horizontal axis). All but DUPS 0 and 1 were eliminated from "merge" including deck 995 complement duplicates (DUPS 13, dark red)(see Examples 1.2). These are all DUPS entries that appeared (in non-trivial numbers in bold) where "CERTAIN" and "UNCERTAIN" pertain to the "WEATHER ELEMENT" certainty (number of weather elements in common and number different):
0 UNIQUE 1 BEST DUPLICATE 4 UNCERTAIN WEATHER ELEMENT 6 TIME/SPACE 8 CERTAIN WEATHER ELEMENT 9 UNCERTAIN WEATHER ELEMENT WITH TIME/SPACE 10 CERTAIN WEATHER ELEMENT WITH TIME/SPACE 11 TIME/SPACE/ID 12 UNCERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID 13 CERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID - DUPS (colors) for the various deck-to-deck matches (horizontal axis) between NCDC and NCEP data. DUPS was 10 for 792-992 matches because of NCEP's ship call sign masking, and 8 for 795-995 matches because of non-identical C-MAN location tables.
- The horizontal axis lists the number of weather elements (W, VV, WW, W1, SLP, AT, SST) in common (zero different), with the bars stratified by DUPS (colors).
Figure 2. (E) DUPS (horizontal axis) of C-MAN-format decks (colors): Group1, IDs in deck 995 only; Group2, IDs in both decks (all deck 795 IDs were Group2). Almost certainly the Group1 deck 995 bars (blue) at DUPS 1 and 13 are the complement duplicates. Regardless, the deck 795 bar (red) at DUPS 1 is almost negligible (see Examples 2.1 for an optimal exception). The deck 795 bar (red) at DUPS 0 (unique) is probably undetected duplicates (see Examples 2.2). (F) Deck 995-995 matches only, number of weather elements in common (horizontal axis) and number different (colors) (see Examples 2.3 for more than zero in common).
Examples 2
These are examples to accompany Fig. 2, in each case showing the IMMA-format reports, followed in the same order by the corresponding input supplementary data (SUPD) for those reports.
- Deck 995 complement duplicates eliminated by dup-merged deck 795 (see Fig. 2E).
A W D A A D
L L T N S P B WP D S T TB B D S U
Y M D H A O ITTLDV I I I CD W V V WW L PI AT BT P S S NCH CC W W W S S S T TS 1 B C I P P
R O Y R T N MCIISS D I D 1I DI WI V W1 PA PT TI TI T I TNHLIHMH D P H D P H I LI 0 1 K D T S
2012 3 1 0 2143 20221 0325 5MOKH1 101767 170 1 245 A 165 234179951141313
2012 3 1 0 2143 20221 0325 5MOKH1 01004 98 0 239 A 165 234179951141313
2012 3 1 0 2143 20221 0325 5MOKH1 01004 98 101767 17 239 1 245 165 2341779510313 1
99 0 CMAN01030120120033 01004 MOKH1 46/// ///// 1//// 40176 57017 90000 222// 00245
99 0 CMAN01030120120104 01004 MOKH1 46/// /1019 10239 90000 333 91223
99 0 CMAN01030120120099 01004 MOKH1 46/// /1019 10239 90000 333 91223 MOKH1 46/// ///// 1//// 40176 57017 90000 222// 00245
- Usage of different location tables currently by NCDC and NCEP, puts C-MAN station with ID SPGF1 in different 1° boxes (B1) resulting in DUPS 0 (unique). C-MAN station with ID FILA2 and longitude 208.005E is not being rounded up by routine "locate."
A W D A A D
L L T N S P B WP D S T TB B D S U
Y M D H A O ITTLDV I I I CD W V V WW L PI AT BT P S S NCH CC W W W S S S T TS 1 B C I P P
R O Y R T N MCIISS D I D 1I DI WI V W1 PA PT TI TI T I TNHLIHMH D P H D P H I LI 0 1 K D T S
2012 3 1 100 2670 28101 0325 5SPGF1 01304 41 102123 50 226 A 165 2426899511413 0
2012 3 1 100 2669 28100 0325 5SPGF1 01304 41 102123 5 226 165 2426979510313 0
99 0 CMAN01030120120120 01014 SPGF1 46/// /1308 10226 40212 53005 90100 333 91209 555 11008 22008 30005 413010 60059 130008 132007 135007 132007 130008 129008
99 0 CMAN01030120120199 01014 SPGF1 46/// /1308 10226 40212 53005 90100 333 91209 555 11008 22008 30005 413010 60059 130008 132007 135007 132007 130008 129008
2012 3 1 0 5933 20800 0325 5FILA2 03004123 100343 8 A 165 1269299511413 0
2012 3 1 0 5933 20801 0325 5FILA2 03004123 100343 8 165 1269179510313 0
99 0 CMAN01030120120017 01004 FILA2 46/// /3024 1//// 40034 53008 90000 333 91228 555 11023 22025 32310 429031 62359 299024 298024 297021 295022 291020 293021
99 0 CMAN01030120120099 01004 FILA2 46/// /3024 1//// 40034 53008 90000 333 91228 555 11023 22025 32310 429031 62359 299024 298024 297021 295022 291020 293021
- Water level stations with IDs TLBO3 and CHYW1 often are "subset" duplicates (i.e. two original GTS messages, one with a subset of meteorological elements represented in the other message) and no elements are lost.
A W D A A D
L L T N S P B WP D S T TB B D S U
Y M D H A O ITTLDV I I I CD W V V WW L PI AT BT P S S NCH CC W W W S S S T TS 1 B C I P P
R O Y R T N MCIISS D I D 1I DI WI V W1 PA PT TI TI T I TNHLIHMH D P H D P H I LI 0 1 K D T S
2012 3 1 130 4556 23609 0325 5TLBO3 02604 51 A 165 165539951141313
2012 3 1 130 4556 23609 0325 5TLBO3 02604 51 10092 0 44 1 77 A 165 1655399511413 1
99 0 CMAN01030120120151 01014 TLBO3 46/// /2610 1//// 90118 333 91215
99 0 CMAN01030120120151 01014 TLBO3 46/// /2610 10044 40092 90118 222// 00077 333 91215
2012 3 1 140 4556 23609 0325 5TLBO3 02404 41 A 165 165539951141313
2012 3 1 140 4556 23609 0325 5TLBO3 02404 41 10092 0 44 1 77 A 165 1655399511413 1
99 0 CMAN01030120120151 01014 TLBO3 46/// /2408 1//// 90124 333 91211
99 0 CMAN01030120120151 01014 TLBO3 46/// /2408 10044 40092 90124 222// 00077 333 91211
2012 3 1 80 4886 23724 0325 5CHYW1 0 604 36 0 18 A 165 165829951141313
2012 3 1 80 4886 23724 0325 5CHYW1 0 604 36 10033 0 18 1 70 A 165 1658299511413 1
99 0 CMAN01030120120151 01014 CHYW1 46/// /0607 10018 90048 333 91210
99 0 CMAN01030120120104 01014 CHYW1 46/// /0607 10018 40033 90048 222// 00070 333 91210
2012 3 1 90 4886 23724 0325 5CHYW1 0 604 36 0 18 A 165 165829951141313
2012 3 1 90 4886 23724 0325 5CHYW1 0 604 36 10035 0 18 1 70 A 165 1658299511413 1
99 0 CMAN01030120120151 01014 CHYW1 46/// /0607 10018 90054 333 91210
99 0 CMAN01030120120120 01014 CHYW1 46/// /0607 10018 40035 90054 222// 00070 333 91210
Figure 3. Dupelim skill, in combining NCDC GTS and NCEP GTS data, for March 2012 (Algorithm 3). As for Algorithm 2 plus NCEP CREX-format stations. As illustrated in Examples 3, NCDC deck 995 provides matches with the CREX transmissions in NCEP deck 797 (NCDC presently does not receive CREX-formatted GTS receipts). The reasons why some e.g. water level station data may be transmitted in the C-MAN format and/or the CREX format are not clearly understood (see also sec. 2). (A) Input (NCDC and NCEP) and output from dupelim ("merge"). (B) Deck composition (colors) of the intermediate output from dupelim for each DUPS (horizontal axis). (C) DUPS (colors) for the various deck-to-deck matches (horizontal axis). (D) Deck 797-995 matches number of weather elements in common (horizontal axis) stratified by DUPS (colors).
Examples 3
These are examples to accompany Fig. 3, in each case showing (1) IMMA reports processed through dupelim, or (2) IMMA-format reports, followed in the same order by the corresponding input supplementary data (SUPD) for those reports.
- DUPS/MATCH = 11 when locations were identical and at least one weather element differed. DUPS/MATCH = 13 water level station with ID PTAW1 complement duplicates matched and one was preferred to, the also incomplete CREX-format report.
MATCH=11 CERTAIN: TIME/SPACE/ID COMMON=3 DIFFER=1
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 4837 23538NEAW1 240 31******100371 22 58******** 70***A*********************************** ****16599511413 1225 504*0** 1*****64
2012 3 1 0 4837 23538NEAW1 239 30******10037**** 58*************************************************** ****165797103**11225 00**************
MATCH=13 CERTAIN: CERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID COMMON=0 DIFFER=0
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 4813 23656PTAW1 ************100291 28 61******** 71***A*********************************** ****16599511413 1225 5***0** 1*****64
2012 3 1 0 4813 23656PTAW1 310 77**********************************A*********************************** ****1659951141313225 504***********64
MATCH=13 CERTAIN: CERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID COMMON=2 DIFFER=0
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 4813 23656PTAW1 ************100291 28 61******** 71***A*********************************** ****16599511413 1225 5***0** 1*****64
2012 3 1 0 4813 23656PTAW1 314 77******10029**** 61*************************************************** ****165797103**13225 00**************
MATCH=13 CERTAIN: CERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID COMMON=1 DIFFER=0
DD WD
L LIIIIIIIII S P W D S S S S IIIII R B D S UU BP
Y M D H A ODDDDDDDDD V WWW L P A B P S NC CC W W W S S S D P HI ERCCCCC RTDV C COO 1 C I P PPTL IDWVITT SHTPWSII
R O Y R T N123456789 D W V W12 PA P T T T TNHLHMH D P H D P H 2 2 2S SS12345 RRSS 1 2SP 0 K D T SCII IIIITII IICBXXXR
2012 3 1 0 4813 23656PTAW1 310 77**********************************A*********************************** ****1659951141313225 504***********64
2012 3 1 0 4813 23656PTAW1 314 77******10029**** 61*************************************************** ****165797103**13225 00**************
- Deck 797-995 matches were undetected when locations were non-identical and at least one weather element differed. Deck 797 wind directions had higher resolution.
A W D A A D
L L T N S P B WP D S T TB B D S U
Y M D H A O ITTLDV I I I CD W V V WW L PI AT BT P S S NCH CC W W W S S S T TS 1 B C I P P
R O Y R T N MCIISS D I D 1I DI WI V W1 PA PT TI TI T I TNHLIHMH D P H D P H I LI 0 1 K D T S
2012 3 1 0 6124 21011 0325 5ANTA2 0 104 31 100244 00 -75 A 165 911999511413 0
2012 3 1 0 6123 21011 0225 0ANTA2 0 14 29 10024 -75 165 9119797103 0
99 0 CMAN01030120120033 01004 ANTA2 46/// /0106 11075 40024 54000 90000 333 91208
99 0 CREX01030120120100 ANTA2 2012 03 01 00 00 00 00 -075 10024 014 0029
2012 3 1 0 4627 27581 0325 5RCKM4 01004 77 100450 0 A 165 1696499511413 0
2012 3 1 0 4626 27581 0225 0RCKM4 0 84 39 10028 165 16964797103 0
99 0 CMAN01030120120033 01004 RCKM4 46/// /1015 1//// 40045 50000 90000 333 91220
99 0 CREX01030120120602 RCKM4 2012 03 01 00 00 00 00 /// 10028 084 0039
2012 3 1 0 4510 27241 0325 5MNMM4 01704 21 99733 100 6 A 165 1695799511413 0
2012 3 1 0 4509 27240 0225 0MNMM4 0312 21 10008 5 165 16957797103 0
99 0 CMAN01030120120017 01004 MNMM4 46/// /1704 10006 49973 53010 90000 333 91205
99 0 CREX01030120120701 MNMM4 2012 03 01 00 00 00 00 005 10008 312 0021
2012 3 1 0 4490 29301 0325 5PSBM1 0 204 62 102360 70 -13 1 42 A 165 1714699511413 0
2012 3 1 0 4490 29302 0225 0PSBM1 0 15 64 10236 -13 165 17146797103 0
99 0 CMAN01030120120033 01004 PSBM1 46/// /0212 11013 40236 50007 90000 222// 00042 333 91214
99 0 CREX01030120120101 PSBM1 2012 03 01 00 00 00 00 -013 10236 015 0064
2012 3 1 0 4439 29179 0325 5ATGM1 03004 10 102334 00 -7 1 24 A 165 1714899511413 0
2012 3 1 0 4440 29180 0225 0ATGM1 0295 8 10233 -7 165 17148797103 0
99 0 CMAN01030120120033 01004 ATGM1 46/// /3002 11007 40233 54000 90000 222// 00024 333 91203
99 0 CREX01030120120101 ATGM1 2012 03 01 00 00 00 00 -007 10233 295 0008
2012 3 1 0 4262 27747 0325 5AGCM4 99958 280 33 A 165 1692299511413 0
2012 3 1 0 4262 27743 0225 0AGCM4 9997 26 165 16922797103 0
99 0 CMAN01030120120033 01004 AGCM4 46/// ///// 10033 49995 58028 90000
99 0 CREX01030120120602 AGCM4 2012 03 01 00 00 00 01 026 09997 /// ////
2012 3 1 0 4159 28859 0325 5QPTR1 0 504 62 101666 300 11 1 49 A 165 1701199511413 0
2012 3 1 0 4158 28859 0225 0QPTR1 0 50 61 10166 11 165 17011797103 0
99 0 CMAN01030120120017 01004 QPTR1 46/// /0512 10011 40166 56030 90000 222// 00049 333 91214
99 0 CREX01030120120101 QPTR1 2012 03 01 00 00 00 00 011 10166 050 0061
2012 3 1 0 3696 28357 0325 5DOMV2 01404 21 101048 270 162 A 165 2066699511413 0
2012 3 1 0 3696 28358 0225 0DOMV2 0143 22 10104 162 165 20666797103 0
99 0 CMAN01030120120017 01004 DOMV2 46/// /1404 10162 40104 58027 90000 333 91205
99 0 CREX01030120120101 DOMV2 2012 03 01 00 00 00 00 162 10104 143 0022
Figure 4. Dupelim skill, in combining NCDC GTS and NCEP GTS data, for March 2012 (Algorithm 4). As for Algorithm 3 plus deck 797 two tenths difference allowance for wind speed and SST. (A) Input (NCDC and NCEP) and output from dupelim ("merge"). (B) Deck composition (colors) of the intermediate output from dupelim for each DUPS (horizontal axis). (C) DUPS (colors) for the various deck-to-deck matches (horizontal axis). (D) Deck 797-995 matches number of weather elements in common (horizontal axis) stratified by DUPS (colors).
Figure 5. Dupelim skill, in combining NCDC GTS and NCEP GTS data, for March 2012 (Algorithm 5). As for Algorithm 4 except deck 797 "space" matches only had to be in the same 1° box, simulating identical location tables. (A) Input (NCDC and NCEP) and output from dupelim ("merge"). (B) Deck composition (colors) of the intermediate output from dupelim for each DUPS (horizontal axis). (C) DUPS (colors) for the various deck-to-deck matches (horizontal axis). (D) Deck 797-995 matches number of weather elements in common (horizontal axis) stratified by DUPS (colors).
[Documentation and Software][Links to additional]
U.S. National Oceanic and Atmospheric Administration hosts the icoads website privacy disclaimer
Document maintained by icoads@noaa.gov
Updated: Mar 27, 2013 22:51:54 UTC
http://www.icoads.noaa.gov/merge_dupelim.html