Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 318.533093ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:25:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 318.565142ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 381.407289ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:25:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 381.430954ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 364.623256ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:24:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 364.643044ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 434.32077ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:24:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 434.340568ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 402.517858ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:24:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 402.542503ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 385.995961ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:23:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 386.014897ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 354.173052ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:23:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 354.193049ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 393.737044ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:22:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 393.764496ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 359.841814ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:21:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 359.851362ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 339.576232ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:21:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 339.600849ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 534.876318ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:21:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 534.892417ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 357.782597ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:21:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 357.82793ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 468.612591ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:20:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 468.633079ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 386.536543ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:20:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 386.554307ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 470.48623ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:20:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 470.530503ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 406.732924ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:19:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 406.750215ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 560.223008ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:19:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 560.236824ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 388.565272ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:18:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 388.599666ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 395.83869ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:18:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 395.855222ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 430.620896ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:17:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 430.634953ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 408.213979ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:17:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 408.235058ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 396.942312ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:17:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 396.958001ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 494.085968ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:16:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 494.105765ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 377.533735ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:16:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 377.547851ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 443.665596ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:16:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 443.690292ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 359.199495ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:16:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 359.253596ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 425.270295ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:15:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 425.292426ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 399.276844ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:15:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 399.296882ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 405.281293ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:14:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 405.30586ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 394.040525ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:14:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 394.068087ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 354.39928ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:14:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 354.418615ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 378.232864ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:13:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 378.247982ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 449.61955ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:13:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 449.633466ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 470.254449ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:12:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 470.279105ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 337.362852ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:12:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 337.374745ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 580.333391ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:11:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 580.357505ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 390.903692ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:11:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 390.972551ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 374.165294ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:10:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 374.187364ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 487.760194ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:10:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 487.775923ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 380.895126ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:10:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 380.919342ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 377.039751ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:09:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 377.179953ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 443.89174ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:09:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 443.911326ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 415.110955ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:09:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 415.133788ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 505.619707ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:08:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 505.696121ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 317.209595ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:08:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 317.232068ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 374.119128ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:07:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 374.135208ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 495.460829ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:07:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 495.476618ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 374.22722ms trace: https://jaeger.dev.storj.io/trace/59fd4cc56b333d27 FAIL (2025-12-21 15:06:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 374.235645ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 496.449315ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:06:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 496.46752ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 396.836828ms trace: https://jaeger.dev.storj.io/trace/d3de09742f8b97b FAIL (2025-12-21 15:05:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments... Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment... Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment. Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece... Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments... Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil> Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments... Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 396.857706ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)