Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
380.500368ms trace:
https://jaeger.dev.storj.io/trace/59fd4cc56b333d27
FAIL (2025-12-21 14:54:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 380.526185ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
356.870414ms trace:
https://jaeger.dev.storj.io/trace/59fd4cc56b333d27
FAIL (2025-12-21 14:54:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 356.888376ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
340.656774ms trace:
https://jaeger.dev.storj.io/trace/d3de09742f8b97b
FAIL (2025-12-21 14:53:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload
Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload
Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments...
Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false
Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment...
Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment.
Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment
Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments...
Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments...
Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 340.688303ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
394.796753ms trace:
https://jaeger.dev.storj.io/trace/59fd4cc56b333d27
FAIL (2025-12-21 14:53:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 394.811651ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
355.348547ms trace:
https://jaeger.dev.storj.io/trace/d3de09742f8b97b
FAIL (2025-12-21 14:53:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload
Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload
Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments...
Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false
Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment...
Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment.
Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment
Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments...
Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments...
Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 355.374465ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
411.146216ms trace:
https://jaeger.dev.storj.io/trace/d3de09742f8b97b
FAIL (2025-12-21 14:52:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload
Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload
Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments...
Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false
Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment...
Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment.
Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment
Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments...
Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments...
Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 411.166354ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
390.989081ms trace:
https://jaeger.dev.storj.io/trace/59fd4cc56b333d27
FAIL (2025-12-21 14:52:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 391.008276ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
393.747814ms trace:
https://jaeger.dev.storj.io/trace/59fd4cc56b333d27
FAIL (2025-12-21 14:52:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 393.78299ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
376.304489ms trace:
https://jaeger.dev.storj.io/trace/d3de09742f8b97b
FAIL (2025-12-21 14:51:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload
Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload
Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments...
Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false
Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment...
Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment.
Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment
Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments...
Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments...
Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 376.32658ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
381.555587ms trace:
https://jaeger.dev.storj.io/trace/59fd4cc56b333d27
FAIL (2025-12-21 14:51:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 381.574032ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
365.563852ms trace:
https://jaeger.dev.storj.io/trace/59fd4cc56b333d27
FAIL (2025-12-21 14:51:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 365.591845ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
398.980568ms trace:
https://jaeger.dev.storj.io/trace/d3de09742f8b97b
FAIL (2025-12-21 14:51:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload
Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload
Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments...
Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false
Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment...
Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment.
Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment
Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments...
Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments...
Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 398.998623ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
389.127951ms trace:
https://jaeger.dev.storj.io/trace/59fd4cc56b333d27
FAIL (2025-12-21 14:50:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 389.146256ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
3.523438493s trace:
https://jaeger.dev.storj.io/trace/d3de09742f8b97b
FAIL (2025-12-21 14:50:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload
Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload
Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments...
Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false
Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment...
Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment.
Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment
Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments...
Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments...
Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 3.523456106s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
315.736261ms trace:
https://jaeger.dev.storj.io/trace/59fd4cc56b333d27
FAIL (2025-12-21 14:49:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 315.75206ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.000725994s trace:
https://jaeger.dev.storj.io/trace/59fd4cc56b333d27
FAIL (2025-12-21 14:49:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000751502s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
6.965281276s trace:
https://jaeger.dev.storj.io/trace/d3de09742f8b97b
FAIL (2025-12-21 14:48:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload
Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload
Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments...
Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false
Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment...
Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment.
Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment
Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments...
Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments...
Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.965298969s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.019186687s trace:
https://jaeger.dev.storj.io/trace/d3de09742f8b97b
FAIL (2025-12-21 14:48:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Dec 21 14:47:50.674657301 (383h18m38.372225617s) [upload=337064]: Starting upload
Dec 21 14:47:50.674708267 (383h18m38.372276573s) [upload=337064]: Done starting upload
Dec 21 14:47:50.674804597 (383h18m38.372372903s) [upload=337064]: Uploading segments...
Dec 21 14:47:50.675034236 (383h18m38.372602552s) [upload=337064]: Got next segment. Inline: false
Dec 21 14:47:50.675063701 (383h18m38.372632017s) [upload=337064]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 21 14:47:50.675068240 (383h18m38.372636546s) [upload=337064]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 21 14:47:50.698765398 (383h18m38.396333714s) [seg_pos={0 0}, upload=337064]: Begin upload segment...
Dec 21 14:47:50.698819038 (383h18m38.396387344s) [seg_pos={0 0}, upload=337064]: Done begin upload segment.
Dec 21 14:47:50.698861047 (383h18m38.396429363s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.698895912 (383h18m38.396464217s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699205750 (383h18m38.396774067s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699448143 (383h18m38.397016449s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.699687842 (383h18m38.397256148s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700000646 (383h18m38.397568952s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700246856 (383h18m38.397815172s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700485371 (383h18m38.398053687s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.700746249 (383h18m38.398314565s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.701774309 (383h18m38.399342615s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.702125746 (383h18m38.399694052s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.703132998 (383h18m38.400701314s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.705128516 (383h18m38.402696821s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.706064504 (383h18m38.40363282s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.708350735 (383h18m38.40591904s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.710549172 (383h18m38.408117478s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Uploading piece...
Dec 21 14:47:50.790883756 (383h18m38.488452072s) [upload=337064]: Next returned nil segment
Dec 21 14:47:50.790892201 (383h18m38.488460507s) [upload=337064]: Waiting for error group managing segments...
Dec 21 14:47:50.798333452 (383h18m38.495901768s) [noise=true , address=145.79.227.49:4337 , piece_id=A647NSUHFN5HGXPUDLIUNRWUBGRLLASEWU5EE6ETVQFBVYWZFXYA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.799747364 (383h18m38.49731567s) [noise=true , address=145.79.227.12:4767 , piece_id=S5KAZYDTZBGP5B3FOY65J35NPVJBC6YUHGN53B3YGIJU6QQAO6TA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.799858922 (383h18m38.497427228s) [noise=true , address=145.79.227.15:4874 , piece_id=7G2KQLRR4LKDH7ZJX6E3OGOOGMI35LCDEJK5LDFRNOZ26EQJUJAA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.801193475 (383h18m38.498761791s) [noise=true , address=145.79.227.18:4006 , piece_id=V4ALIQPBUZKZEWE7DOVNBCM66ONKOM4KJMPDICRESQUA7DQIZCQA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.802050296 (383h18m38.499618612s) [noise=true , address=145.79.227.50:4374 , piece_id=3BZWJVDGVCMD5XJB2WE64DQ4GNJQ6W35W6OZSHYW4AW3ENWFZ4FQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.803782973 (383h18m38.501351289s) [noise=true , address=145.79.227.35:4482 , piece_id=Y3CLJPNVZX5DJSMPBQZ6C3E62WX34PPZ2SFDXAXALF3SHZZFWNMQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.822261843 (383h18m38.519830149s) [noise=true , address=45.91.116.10:4084 , piece_id=AUQ5YSVYBB33Z27KAJ57VQEWRMA2OGJVH3LJE7SW7KX2SCB6KSNA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.838166866 (383h18m38.535735182s) [noise=true , address=45.91.116.9:4263 , piece_id=TSFD5TQ2AJPKIYI5UBRTNHHUSWWQ6SQGCFQSL4WEVBGWXFVMS43A, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.859757239 (383h18m38.557325545s) [noise=true , address=77.90.31.10:4517 , piece_id=SXYATEOI4RF3Y64MOQGZMWNVWXG7YKAQ7WBWNWX7J3GU5ZDY77CQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:50.866747007 (383h18m38.564315323s) [noise=true , address=45.91.116.7:4009 , piece_id=DMQAUNKL65SX4RSQJA6AMHCEWE7IJMTZDNUAXMZOWXWOBSN5KVDA, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:47:51.021403174 (383h18m38.71897149s) [noise=true , address=45.91.116.14:4191 , piece_id=O57XNUHHSB7RCEL66W3IDALNCLNJZHQU4MZ2PCA2AADGG6RG6YUQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: <nil>
Dec 21 14:48:20.661720038 (383h19m8.359288383s) [noise=true , address=145.79.227.25:4617 , piece_id=6477E2Z5P2J26XQ3PBXBIEGH4RFBH2N27X26UY52V34EU5PUUAOQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.661824302 (383h19m8.359392648s) [noise=true , address=145.79.227.8:4330 , piece_id=ZGYE3M3VIB5X6BDZBOMCX3QYVQZTNWVCNOJG7DQAKNCOUNG3DCJQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662099496 (383h19m8.359667903s) [noise=true , address=145.79.227.38:4977 , piece_id=FEPK5PJVHJUTPQWXFM4KCIHJO3QYWKNKO4EUCOZI4VQN2UNMNVCQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662138559 (383h19m8.359706896s) [noise=true , address=145.79.227.29:4943 , piece_id=6TA446AU6WJLRHA23L77NB4UTNR5XHATTQ5LKJ3LZ6ZBKDE54MAQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662304589 (383h19m8.359872946s) [noise=true , address=145.79.227.41:4086 , piece_id=TSYZ6GWSGF5WPK4RDK26FWA3APXODFDTGBYM4DWKLOYISXM3X5WQ, seg_pos={0 0}, upload=337064]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF
Dec 21 14:48:20.662344654 (383h19m8.359912989s) [seg_pos={0 0}, upload=337064]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Dec 21 14:48:20.678654322 (383h19m8.376222658s) [upload=337064]: Done uploading segments...
Dec 21 14:48:20.678678688 (383h19m8.376247024s) [upload=337064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.019212315s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
4.453486575s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:53:15 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.453507574s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.444512463s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:52:27 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 5.44453242s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
4.390614583s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:51:59 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.390638948s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
4.59755156s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:51:42 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.597573582s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
4.640328497s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:50:32 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.640349897s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
4.470096006s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:50:14 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.47011988s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
4.55985492s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:49:24 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.559875108s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
4.585327659s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:48:19 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.585350582s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
4.352531488s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:47:37 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.352553008s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
4.577156007s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:47:15 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.577176214s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
4.894285409s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:46:12 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.894305005s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
4.675844869s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:45:27 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.675866911s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
4.647859192s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:44:28 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.647883298s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
4.484056804s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:43:40 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.48408667s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
4.420801664s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:42:25 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.42082595s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
5.16523197s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:41:41 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 5.16525978s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
4.617413421s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:41:08 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.617439301s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
4.623876439s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:40:11 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.623900303s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
4.578582851s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:39:40 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.578607707s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
4.370383922s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:39:16 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.370438845s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
4.393699802s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:38:12 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.393722375s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
4.575796557s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:37:21 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.575817045s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
4.407580072s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:36:46 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.40760051s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
4.355407961s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:35:53 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.355431987s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
4.570529125s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:34:46 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.570551428s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
4.552127058s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:33:44 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.552152817s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
4.226592161s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:32:38 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.226615035s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
4.771656247s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:31:59 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.771678027s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
4.67807481s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:31:03 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.678097252s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
4.406041356s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:30:41 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.406068817s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
4.869120827s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:29:28 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.869141477s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
4.479479933s trace:
https://jaeger.dev.storj.io/trace/4d7f8384257e3d28
FAIL (2025-12-21 11:28:13 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.479507394s
core
uplink
ap1
|