Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 470.907514ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:30:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.925568ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 338.831666ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:29:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 338.849109ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 363.59243ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:29:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 363.612908ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 452.585889ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:28:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 452.604354ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 470.441961ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:28:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.463953ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 461.202594ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:27:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 461.220819ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 425.419622ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:27:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 425.444176ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 383.268575ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:26:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 383.288ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 538.281378ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:26:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 538.303821ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 347.98858ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:25:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 348.005502ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 459.721109ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:24:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 459.738561ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 568.369475ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:24:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 568.389531ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 361.72485ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:22:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 361.740309ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 375.254794ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:22:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 375.272918ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 441.56546ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:21:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 441.583342ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 439.202929ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:19:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 439.222976ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 390.555145ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:18:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 390.577567ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 361.111522ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:17:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 361.129457ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 329.552704ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:17:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 329.570748ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 439.208317ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:16:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 439.2528ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 478.603746ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:15:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 478.622501ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 497.600761ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:15:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 497.617442ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 397.877843ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:14:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 397.896869ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 455.428646ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:13:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.445628ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 453.849985ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:12:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 453.866646ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 400.621959ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:11:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 400.641966ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 413.215442ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:10:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 413.227324ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 387.932465ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:10:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 387.948355ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 358.677828ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:09:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 358.697493ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 455.14733ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:08:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.178991ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 417.9306ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:07:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 417.949796ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 342.266172ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:06:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 342.284746ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 466.102681ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:05:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 466.118783ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.034823074s trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:05:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.034851147s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 374.320183ms trace: https://jaeger.dev.storj.io/trace/6295afa6d8d7f95 FAIL (2025-12-06 09:14:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 374.346914ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 324.637696ms trace: https://jaeger.dev.storj.io/trace/6295afa6d8d7f95 FAIL (2025-12-06 09:13:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 324.65588ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 325.298792ms trace: https://jaeger.dev.storj.io/trace/6295afa6d8d7f95 FAIL (2025-12-06 09:12:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 325.317416ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 346.136928ms trace: https://jaeger.dev.storj.io/trace/6295afa6d8d7f95 FAIL (2025-12-06 09:11:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 346.176652ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 330.386254ms trace: https://jaeger.dev.storj.io/trace/6295afa6d8d7f95 FAIL (2025-12-06 09:11:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 330.402925ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 462.758868ms trace: https://jaeger.dev.storj.io/trace/6295afa6d8d7f95 FAIL (2025-12-06 09:10:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 462.77599ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 342.607886ms trace: https://jaeger.dev.storj.io/trace/6295afa6d8d7f95 FAIL (2025-12-06 09:09:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 342.651158ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 373.08027ms trace: https://jaeger.dev.storj.io/trace/6295afa6d8d7f95 FAIL (2025-12-06 09:08:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 373.100056ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 339.575893ms trace: https://jaeger.dev.storj.io/trace/6295afa6d8d7f95 FAIL (2025-12-06 09:08:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 339.604898ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 443.149861ms trace: https://jaeger.dev.storj.io/trace/6295afa6d8d7f95 FAIL (2025-12-06 09:07:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 443.167163ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 322.023873ms trace: https://jaeger.dev.storj.io/trace/6295afa6d8d7f95 FAIL (2025-12-06 09:06:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 322.043579ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 341.152518ms trace: https://jaeger.dev.storj.io/trace/6295afa6d8d7f95 FAIL (2025-12-06 09:05:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 341.169629ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 349.510138ms trace: https://jaeger.dev.storj.io/trace/6295afa6d8d7f95 FAIL (2025-12-06 09:05:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 349.534205ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 362.554795ms trace: https://jaeger.dev.storj.io/trace/6295afa6d8d7f95 FAIL (2025-12-06 09:04:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 362.575785ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 372.343302ms trace: https://jaeger.dev.storj.io/trace/6295afa6d8d7f95 FAIL (2025-12-06 09:03:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 372.368489ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 331.942311ms trace: https://jaeger.dev.storj.io/trace/6295afa6d8d7f95 FAIL (2025-12-06 09:02:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 331.954162ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)