Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 406.612717ms trace: https://jaeger.dev.storj.io/trace/617f3983448445c3 FAIL (2025-12-08 14:26:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 406.629227ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 347.831509ms trace: https://jaeger.dev.storj.io/trace/617f3983448445c3 FAIL (2025-12-08 14:25:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 347.852058ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 367.680503ms trace: https://jaeger.dev.storj.io/trace/617f3983448445c3 FAIL (2025-12-08 14:24:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 367.702574ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 347.591399ms trace: https://jaeger.dev.storj.io/trace/617f3983448445c3 FAIL (2025-12-08 14:23:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 347.61395ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 346.015123ms trace: https://jaeger.dev.storj.io/trace/617f3983448445c3 FAIL (2025-12-08 14:22:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 346.038308ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 396.751298ms trace: https://jaeger.dev.storj.io/trace/617f3983448445c3 FAIL (2025-12-08 14:21:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 396.79567ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 395.214236ms trace: https://jaeger.dev.storj.io/trace/617f3983448445c3 FAIL (2025-12-08 14:20:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 395.230065ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 398.28937ms trace: https://jaeger.dev.storj.io/trace/617f3983448445c3 FAIL (2025-12-08 14:20:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 398.307012ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 437.432448ms trace: https://jaeger.dev.storj.io/trace/617f3983448445c3 FAIL (2025-12-08 14:19:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 437.448558ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 387.50778ms trace: https://jaeger.dev.storj.io/trace/617f3983448445c3 FAIL (2025-12-08 14:19:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 387.520234ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 402.504026ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 06:11:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 402.524053ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 398.647211ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 06:11:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 398.668852ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 366.010633ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 06:09:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 366.023057ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 352.50409ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 06:09:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 352.522183ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 565.393792ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 06:08:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 565.412777ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 360.773459ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 06:08:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 360.805671ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 382.824403ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 06:06:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 382.842056ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 417.36715ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 06:05:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 417.387718ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 476.324529ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 06:04:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 476.342913ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 384.700019ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 06:03:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 384.717221ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 385.095697ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 06:03:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 385.112067ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 460.402077ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 06:02:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 460.423406ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 452.305811ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 06:01:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 452.325338ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 436.470817ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 06:00:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 436.53141ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 347.388836ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:59:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 347.409205ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 418.463191ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:59:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 418.486154ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 483.693426ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:58:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 483.711501ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 363.904529ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:57:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 363.920028ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 370.737003ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:57:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 370.754113ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 422.735781ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:56:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 422.750629ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 327.318253ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:55:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 327.339121ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 377.012346ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:55:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 377.060284ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 463.142984ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:54:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 463.157483ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 396.903812ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:53:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 396.922337ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 398.350631ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:52:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 398.367753ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 376.031991ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:51:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 376.056899ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 334.419147ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:51:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 334.435998ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 324.585967ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:50:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 324.605282ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 430.088917ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:50:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 430.125985ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 411.827321ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:49:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 411.846537ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 421.389576ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:48:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 421.411175ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 392.163619ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:47:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 392.183126ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 403.520786ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:46:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 403.541805ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 336.012931ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:45:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 336.030053ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 421.802356ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:44:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 421.821041ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 343.0509ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:43:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 343.078513ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 421.785381ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:42:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 421.806178ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 382.73232ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:41:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 382.748991ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 443.787146ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:41:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 443.806042ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 400.19413ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:40:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 400.214529ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)