Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 416.0881ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:39:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 416.1092ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 402.916786ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:39:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 402.932276ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 435.422801ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:38:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 435.442738ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 370.555732ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:37:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 370.573615ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 411.685021ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:37:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 411.707383ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 368.346913ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:36:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 368.364477ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 445.123745ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:35:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 445.146178ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 377.08677ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:34:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 377.100897ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 346.781846ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:34:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 346.800572ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 354.043014ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:32:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 354.059425ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 368.795093ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:32:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 368.816183ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 341.171294ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:31:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 341.193015ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 394.717208ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:30:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 394.744187ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 413.189958ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:30:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 413.205147ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 463.571772ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:29:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 463.595607ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 337.042953ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:28:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 337.060627ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 367.74937ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:28:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 367.767955ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 459.896755ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:27:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 459.9147ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 448.535189ms trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:27:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 448.560988ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.015885221s trace: https://jaeger.dev.storj.io/trace/54e36c20f1dd6c59 FAIL (2025-12-08 05:26:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 05:26:23.924358641 (61h57m11.621926947s) [upload=54303]: Starting upload Dec 8 05:26:23.924400298 (61h57m11.621968614s) [upload=54303]: Done starting upload Dec 8 05:26:23.924465821 (61h57m11.622034127s) [upload=54303]: Uploading segments... Dec 8 05:26:23.924642451 (61h57m11.622210757s) [upload=54303]: Got next segment. Inline: false Dec 8 05:26:23.924670473 (61h57m11.622238789s) [upload=54303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:26:23.924674069 (61h57m11.622242385s) [upload=54303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 05:26:23.942221161 (61h57m11.639789477s) [seg_pos={0 0}, upload=54303]: Begin upload segment... Dec 8 05:26:23.942261125 (61h57m11.639829441s) [seg_pos={0 0}, upload=54303]: Done begin upload segment. Dec 8 05:26:23.942304387 (61h57m11.639872703s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942341306 (61h57m11.639909612s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.942682202 (61h57m11.640250518s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943047405 (61h57m11.64061571s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943300397 (61h57m11.640868713s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943695675 (61h57m11.641263981s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.943986128 (61h57m11.641554434s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944375235 (61h57m11.641943542s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.944666780 (61h57m11.642235096s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.945868975 (61h57m11.643437282s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.946924698 (61h57m11.644493014s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.948059267 (61h57m11.645627584s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.949702067 (61h57m11.647270393s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.951583091 (61h57m11.649151428s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.953870314 (61h57m11.65143865s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.954342065 (61h57m11.651910381s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Uploading piece... Dec 8 05:26:23.988883304 (61h57m11.68645162s) [noise=true , address=77.90.31.21:4879 , piece_id=SB4KQXMZNAX6ZVUFT4FP5VBTDZAIUWPND27AN2OE33IXXCHCEPOQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:23.995100049 (61h57m11.692668354s) [noise=true , address=77.90.31.19:4841 , piece_id=X6KARONIFVZMYTUU3V42ZDDMIWPYHLE2AXJ4AOCHUGXEXQ3SUHVQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 05:26:24.033065424 (61h57m11.73063373s) [upload=54303]: Next returned nil segment Dec 8 05:26:24.033073760 (61h57m11.730642066s) [upload=54303]: Waiting for error group managing segments... Dec 8 05:26:24.044708365 (61h57m11.74227667s) [noise=true , address=145.79.227.14:4837 , piece_id=FR6RUT5LK5DMF7FNODX7WQOUZZR7Y6X6GBMR6NNW37QPEI3EGZIQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045148557 (61h57m11.742716863s) [noise=true , address=145.79.227.31:4797 , piece_id=UIPM363TXOOZAUZFDFWUFZI5S7PL4ITDOHPS7M4RYLOBE33DWYSQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045191438 (61h57m11.742759754s) [noise=true , address=145.79.227.32:4836 , piece_id=PQLSQIDEBMRAFSZBKCPJTVSIAEXG7NLE437MNKXIMFC6UZQPHPCA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045530310 (61h57m11.743098606s) [noise=true , address=145.79.227.21:4474 , piece_id=V62BHKG6TY2DZCB6UOMRQJMMQVMABOIXFZ2TEVLAHQUIDQTQTY2Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.045867901 (61h57m11.743436217s) [noise=true , address=145.79.227.22:4509 , piece_id=PPODMYHFVW4ICA4BH3LDMGH2OPCVEQ6RKLIHDPY3CJESEC3IEPVA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046012722 (61h57m11.743581038s) [noise=true , address=145.79.227.11:4731 , piece_id=TV5ACR4OLCIR6D2FF2RAIWE5U4LGBKCVW4U2U2KBWHHV5QDUSWXA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046272426 (61h57m11.743840733s) [noise=true , address=145.79.227.23:4547 , piece_id=CZHAL4T6OAKWA657NAFNOMXGRCIYEZ3EPTAZU7KAUAHD7A6QAL3Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.046725784 (61h57m11.7442941s) [noise=true , address=145.79.227.28:4725 , piece_id=LSMS6CARZ3AE5OONZTEPMTQS66BVGY7PCRU6UBK7R3M7JMM6VEDA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.047503135 (61h57m11.745071451s) [noise=true , address=145.79.227.27:4689 , piece_id=XTXQFU6SCL737UWFMNLRKLHM56TUK2ZZ3P5OUA4ROWYD7HORXFFA, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.051681821 (61h57m11.749250137s) [noise=true , address=145.79.227.26:4655 , piece_id=3U7NN7JNVDYQKTAYO5YMAQCWIKK3Q7BRWVTBYNTMTAHNV4YKW2TQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.055289332 (61h57m11.752857637s) [noise=true , address=77.90.31.19:4843 , piece_id=Z63ZW4ENKV5TPIESX4HDFGRZFR4YSMBEOBMUY3L4DB5RWLAMK52Q, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:24.075195039 (61h57m11.772763344s) [noise=true , address=77.90.31.14:4662 , piece_id=TEPIXZU37PUUBHNJTUKLUVHJIX4EGVMEJUJ73GQWTIS75MFKGTFQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: <nil> Dec 8 05:26:53.910998538 (61h57m41.608566854s) [noise=true , address=77.90.31.16:4734 , piece_id=3CM5NJEWCCDL6A6U7CPWGLTQMYC7P3MQ3PXFPLQFYU5NNXTVDPUQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911079619 (61h57m41.608647935s) [noise=true , address=77.90.31.16:4735 , piece_id=BEYXS5SNT7XKJFXI7XUOEJD2OKN7Z2PD5LR6SVVVWVYKQBNJ34NQ, seg_pos={0 0}, upload=54303]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 05:26:53.911107141 (61h57m41.608675457s) [seg_pos={0 0}, upload=54303]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 05:26:53.924986570 (61h57m41.622554876s) [upload=54303]: Done uploading segments... Dec 8 05:26:53.925005475 (61h57m41.622573792s) [upload=54303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.015901483s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 397.601308ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:50:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 397.617077ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 412.658844ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:50:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 412.672049ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 358.105477ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:49:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 358.127067ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 430.110855ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:48:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 430.132024ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 366.15328ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:47:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 366.170173ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 334.658908ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:47:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 334.67583ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 332.854507ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:46:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 332.871789ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 413.672507ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:45:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 413.685971ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 394.575834ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:44:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 394.592362ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 360.511541ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:43:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 360.554983ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 335.440369ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:43:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 335.45215ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 353.93542ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:42:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 353.95056ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 350.078876ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:41:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 350.099304ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 378.004166ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:41:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 378.04366ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 357.569617ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:40:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 357.586909ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 408.146713ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:38:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 408.161341ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 385.829812ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:38:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 385.862072ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 485.103505ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:37:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 485.126619ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 398.751971ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:36:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 398.77255ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 400.48006ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:35:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 400.526457ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 394.569035ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:34:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 394.593322ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 380.039192ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:33:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 380.054852ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 380.733705ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:33:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 380.753401ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 374.591898ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:32:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 374.622766ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 354.799565ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:31:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 354.817158ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 406.2964ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:30:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 406.313522ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 354.441955ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:29:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 354.464607ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 345.035214ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:28:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 345.051493ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 384.577695ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:28:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 384.599194ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 349.429312ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:27:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 349.446494ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)