Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 444.261463ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:43:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 444.28196ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 360.843109ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:42:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 360.861383ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 428.84433ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:41:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 428.870889ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 443.178265ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:40:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 443.198983ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 470.700277ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:39:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.721246ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 550.159806ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:38:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 550.179413ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 558.972718ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:37:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 558.996412ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 431.365119ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:37:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 431.385766ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 416.042156ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:36:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 416.179673ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 452.595758ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:36:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 452.606006ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 419.88228ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:35:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 419.943775ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 465.778353ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:34:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 465.80333ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 422.721703ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:34:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 422.739014ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 327.090515ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:33:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 327.107016ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 398.283431ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:32:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 398.302397ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 335.405807ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:32:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 335.425424ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 373.908511ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:31:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 373.919541ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 373.921574ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:31:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 373.952662ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 470.907514ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:30:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.925568ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 338.831666ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:29:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 338.849109ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 363.59243ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:29:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 363.612908ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 452.585889ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:28:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 452.604354ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 470.441961ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:28:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.463953ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 461.202594ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:27:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 461.220819ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 425.419622ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:27:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 425.444176ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 383.268575ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:26:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 383.288ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 538.281378ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:26:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 538.303821ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 347.98858ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:25:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 348.005502ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 459.721109ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:24:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 459.738561ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 568.369475ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:24:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 568.389531ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 361.72485ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:22:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 361.740309ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 375.254794ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:22:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 375.272918ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 441.56546ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:21:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 441.583342ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 439.202929ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:19:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 439.222976ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 390.555145ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:18:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 390.577567ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 361.111522ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:17:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 361.129457ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 329.552704ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:17:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 329.570748ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 439.208317ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:16:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 439.2528ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 478.603746ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:15:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 478.622501ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 497.600761ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:15:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 497.617442ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 397.877843ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:14:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 397.896869ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 455.428646ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:13:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.445628ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 453.849985ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:12:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 453.866646ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 400.621959ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:11:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 400.641966ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 413.215442ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:10:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 413.227324ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 387.932465ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:10:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 387.948355ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 358.677828ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:09:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 358.697493ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 455.14733ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:08:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.178991ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 417.9306ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:07:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 417.949796ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 342.266172ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:06:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 342.284746ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)