Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 405.531878ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:20:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 405.551226ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 459.362236ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:19:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 459.382274ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX............................................] duration: 383.952304ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:19:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 383.973774ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 436.606617ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:18:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 436.619792ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX...........................................] duration: 510.896757ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:18:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 510.91446ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX..........................................] duration: 520.029557ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:17:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 520.063551ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 423.029123ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:17:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 423.046956ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 389.086216ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-11 00:17:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 389.101654ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.........................................] duration: 433.309968ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:16:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 433.33214ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 441.677097ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:16:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 441.704678ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 359.717259ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-11 00:16:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 359.739812ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.X.........................................................] duration: 366.155857ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-11 00:15:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 366.174922ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 421.767684ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:15:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 421.782441ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX........................................] duration: 508.511539ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:15:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 508.530664ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 447.054037ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:15:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 447.078302ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.X........................................................] duration: 396.382661ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-11 00:15:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 396.404262ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.......................................] duration: 456.522511ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:15:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 456.540074ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX......................................] duration: 417.073134ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:14:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 417.093061ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.X.......................................................] duration: 377.258767ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-11 00:14:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 377.276629ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 435.493725ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:14:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 435.512ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.....................................] duration: 522.451951ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:14:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 522.46778ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.X......................................................] duration: 383.075334ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-11 00:14:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 383.098286ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX....................................] duration: 386.042266ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:13:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 386.060289ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.X.....................................................] duration: 353.316893ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-11 00:13:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 353.337212ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 475.615416ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:13:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 475.631585ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX...................................] duration: 563.522252ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:13:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 563.544724ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 441.436756ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:12:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 441.458237ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X....X.X....................................................] duration: 364.367837ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-11 00:12:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 364.383587ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX..................................] duration: 544.625453ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:12:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 544.642563ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X....X.X...................................................] duration: 573.930525ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-11 00:12:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 573.954039ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 528.650779ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:11:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 528.667501ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.................................] duration: 463.04836ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:11:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 463.06479ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.X....X.X..................................................] duration: 356.585699ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-11 00:11:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 356.646643ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXX................................] duration: 392.264351ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:10:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 392.282663ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 412.22651ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:10:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 412.245256ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXXXXXXXXXX...............................] duration: 516.126323ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:10:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 516.185514ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.X....X.X.................................................] duration: 354.943901ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-11 00:10:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 354.971282ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXXXXXXXXXXX..............................] duration: 430.46952ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:10:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 430.481613ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.X....X.X................................................] duration: 386.909598ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-11 00:09:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 386.926198ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 360.448847ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:09:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 360.460198ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..X.X....X.X...............................................] duration: 361.495191ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-11 00:09:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 361.52083ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXXXXXXXXXX.............................] duration: 578.103702ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:09:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 578.131634ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 467.518812ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:08:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.537256ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [.....XXXXXXXXXXXXXXXXXXXXXXXXXXX............................] duration: 544.716905ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:08:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 544.856435ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..X.X....X.X..............................................] duration: 341.781944ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-11 00:08:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 341.969164ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 508.238662ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-11 00:08:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 508.263528ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXXXXXXXXX...........................] duration: 385.803731ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:08:00 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 385.817876ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 551.782672ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:07:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 551.800986ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..X.X....X.X.............................................] duration: 340.2959ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-11 00:07:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 340.306951ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [XX..........................................................] duration: 713.344603ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-11 00:07:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 713.361726ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)