Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....................X......................X..X.XXX...XX.XX] duration: 30.000884242s FAIL (2025-09-03 20:01:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000905201s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................................XX.X..X.X..X......] duration: 490.0394ms FAIL (2025-09-03 20:01:42 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 490.064848ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......................X......................X..X.XXX...XX.X] duration: 30.000683284s FAIL (2025-09-03 20:01:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000701859s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................................XX.X..X.X..X.....] duration: 400.994064ms FAIL (2025-09-03 20:01:08 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 401.016056ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................................XX.X..X.X..X....] duration: 494.194094ms FAIL (2025-09-03 19:59:59 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 494.208932ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................X......................X..X.XXX...XX.] duration: 1.109706576s FAIL (2025-09-03 19:59:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.109724871s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................X......................X..X.XXX...XX] duration: 30.001207128s FAIL (2025-09-03 19:59:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001225963s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................................XX.X..X.X..X...] duration: 509.826041ms FAIL (2025-09-03 19:59:14 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 509.885984ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........................X......................X..X.XXX...X] duration: 30.000506643s FAIL (2025-09-03 19:58:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000526171s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................................XX.X..X.X..X..] duration: 539.260219ms FAIL (2025-09-03 19:58:04 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 539.281637ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................................XX.X..X.X..X.] duration: 328.396203ms FAIL (2025-09-03 19:57:16 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 328.408777ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................................XX.X..X.X..X] duration: 30.015395107s FAIL (2025-09-03 19:57:15 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.015418341s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................X......................X..X.XXX...] duration: 522.667021ms FAIL (2025-09-03 19:56:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 522.690654ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................X......................X..X.XXX..] duration: 486.301631ms FAIL (2025-09-03 19:56:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 486.32182ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................XX.X..X.X..] duration: 542.277554ms FAIL (2025-09-03 19:55:57 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 19:54:14.403115658 (125h45m49.755900787s) [upload=110885]: Starting upload Sep 3 19:54:14.403199486 (125h45m49.755984625s) [upload=110885]: Done starting upload Sep 3 19:54:14.403287271 (125h45m49.756072399s) [upload=110885]: Uploading segments... Sep 3 19:54:14.403649492 (125h45m49.75643462s) [upload=110885]: Got next segment. Inline: false Sep 3 19:54:14.403697302 (125h45m49.75648243s) [upload=110885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:54:14.403703914 (125h45m49.756489042s) [upload=110885]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:54:14.497238203 (125h45m49.850023331s) [seg_pos={0 0}, upload=110885]: Begin upload segment... Sep 3 19:54:14.497280222 (125h45m49.85006535s) [seg_pos={0 0}, upload=110885]: Done begin upload segment. Sep 3 19:54:14.497311981 (125h45m49.85009711s) [noise=true , address=104.36.82.16:4295 , piece_id=UMMOFFB3TF5W64MBSHSAFTNELPDTX36ZK4LQAXMXCF25GGLZBQQA, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.497817503 (125h45m49.85060263s) [noise=true , address=104.36.82.5:4838 , piece_id=KKILOVJ7AXF3O2FXGZD3COVNQWOYACCFWO3WUGPXXOQFZAS4WQ3Q, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.498182630 (125h45m49.850967757s) [noise=true , address=104.36.82.14:4222 , piece_id=D73WUVHJKYLLGRWGJEEVKEOPQYUAKJWNLLRKQAX3XPZZZDAI5QOQ, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.498550361 (125h45m49.851335489s) [noise=true , address=77.90.31.15:4698 , piece_id=PM64UCH6ICOKWW3MN4Z6J7E76VCQLF7NT4XQRPIRUZPAQ2LCVY4A, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.498942198 (125h45m49.851727336s) [noise=true , address=45.91.116.9:4262 , piece_id=NICM4KJ6KL4XFDF3M24UDPYIX2CA7VO62TOYNVDV5T5C34TMFNVA, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.499290594 (125h45m49.852075721s) [noise=true , address=45.91.116.13:4153 , piece_id=OIXRMOJZAXNBDJH4Y4ZMR2GJTPTBBMFMHUHYAHII3NZIZPLQVTYA, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.499664365 (125h45m49.852449503s) [noise=true , address=45.91.116.6:4975 , piece_id=76BSHCO66KXF7VXQAEJEAWNM7LMJY3VIOQSOVF2TIZUAMT3GZIJQ, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.499800581 (125h45m49.85258572s) [noise=true , address=45.91.116.11:4117 , piece_id=OZW5SRUEBR3HVB5OPPBLQMHFMNDSM3T4ADX5OXDUB5S3MGUWBG2Q, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.501211505 (125h45m49.853996633s) [noise=true , address=45.91.116.5:4299 , piece_id=SOQG4YIAAESZFZHQ4TPSZ6HKLOGELLDG5PCYJVN7C6CNQ5HYWJQA, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.501780004 (125h45m49.854565132s) [noise=true , address=104.36.82.2:4729 , piece_id=VRETJR2UUWHEGSO5N32SBCECAUYU2RGSQJCOPV7WPRLVFS6M5YRQ, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.502538641 (125h45m49.855323769s) [noise=true , address=77.90.31.21:4877 , piece_id=BFN2K35F33QD53MCE2C75UEDC7Q2HYEJOKVU6HYNUTRE6BKJJCFA, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.502975021 (125h45m49.85576015s) [noise=true , address=45.91.116.4:4940 , piece_id=R2NOHO4FHKJP4FRDIAB6DFRSUFTAUJJKFJW753SASSKRMKQMLX5Q, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.503470433 (125h45m49.856255562s) [noise=true , address=45.91.116.13:4154 , piece_id=4S4477Z5QVN3TMWKPILSNRBJA3OJ23YGVK36I5HDUO5IHJVSJB5A, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.503900312 (125h45m49.85668544s) [noise=true , address=45.91.116.11:4119 , piece_id=Q55DIOGQJKGXASB2OVAWPBHIY5MFL76EG5F6SIRCJPYETW4SWD3Q, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.504301726 (125h45m49.857086855s) [noise=true , address=45.91.116.6:4974 , piece_id=RZIN6NGRPAYUXXAYBTDJW6V5OVKQBJ7IQBUK5OEVIRZ53WIFRHYA, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.504666874 (125h45m49.857452001s) [noise=true , address=45.91.116.7:4011 , piece_id=YNJ4YFR67GYFCBNCO4FL5W3P2OTU4NPCZKHIIQEWIT2Y7HULPK5Q, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.614557620 (125h45m49.967342747s) [upload=110885]: Next returned nil segment Sep 3 19:54:14.614573099 (125h45m49.967358227s) [upload=110885]: Waiting for error group managing segments... Sep 3 19:54:14.623997493 (125h45m49.976782621s) [noise=true , address=45.91.116.11:4119 , piece_id=Q55DIOGQJKGXASB2OVAWPBHIY5MFL76EG5F6SIRCJPYETW4SWD3Q, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.624439564 (125h45m49.977224692s) [noise=true , address=45.91.116.11:4117 , piece_id=OZW5SRUEBR3HVB5OPPBLQMHFMNDSM3T4ADX5OXDUB5S3MGUWBG2Q, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.624590588 (125h45m49.977375726s) [noise=true , address=45.91.116.13:4153 , piece_id=OIXRMOJZAXNBDJH4Y4ZMR2GJTPTBBMFMHUHYAHII3NZIZPLQVTYA, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.625540105 (125h45m49.978325232s) [noise=true , address=45.91.116.13:4154 , piece_id=4S4477Z5QVN3TMWKPILSNRBJA3OJ23YGVK36I5HDUO5IHJVSJB5A, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.633640358 (125h45m49.986425476s) [noise=true , address=45.91.116.9:4262 , piece_id=NICM4KJ6KL4XFDF3M24UDPYIX2CA7VO62TOYNVDV5T5C34TMFNVA, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.633790400 (125h45m49.986575517s) [noise=true , address=45.91.116.6:4974 , piece_id=RZIN6NGRPAYUXXAYBTDJW6V5OVKQBJ7IQBUK5OEVIRZ53WIFRHYA, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.635124730 (125h45m49.987909847s) [noise=true , address=45.91.116.4:4940 , piece_id=R2NOHO4FHKJP4FRDIAB6DFRSUFTAUJJKFJW753SASSKRMKQMLX5Q, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.635626583 (125h45m49.988411702s) [noise=true , address=45.91.116.6:4975 , piece_id=76BSHCO66KXF7VXQAEJEAWNM7LMJY3VIOQSOVF2TIZUAMT3GZIJQ, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.636979518 (125h45m49.989764626s) [noise=true , address=45.91.116.5:4299 , piece_id=SOQG4YIAAESZFZHQ4TPSZ6HKLOGELLDG5PCYJVN7C6CNQ5HYWJQA, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.641360187 (125h45m49.994145315s) [noise=true , address=45.91.116.7:4011 , piece_id=YNJ4YFR67GYFCBNCO4FL5W3P2OTU4NPCZKHIIQEWIT2Y7HULPK5Q, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.649885590 (125h45m50.002670708s) [noise=true , address=77.90.31.15:4698 , piece_id=PM64UCH6ICOKWW3MN4Z6J7E76VCQLF7NT4XQRPIRUZPAQ2LCVY4A, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.722385248 (125h45m50.075170376s) [noise=true , address=77.90.31.21:4877 , piece_id=BFN2K35F33QD53MCE2C75UEDC7Q2HYEJOKVU6HYNUTRE6BKJJCFA, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:44.388239984 (125h46m19.741025092s) [noise=true , address=104.36.82.5:4838 , piece_id=KKILOVJ7AXF3O2FXGZD3COVNQWOYACCFWO3WUGPXXOQFZAS4WQ3Q, seg_pos={0 0}, upload=110885]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:54:44.388282233 (125h46m19.741067341s) [noise=true , address=104.36.82.16:4295 , piece_id=UMMOFFB3TF5W64MBSHSAFTNELPDTX36ZK4LQAXMXCF25GGLZBQQA, seg_pos={0 0}, upload=110885]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:54:44.388314784 (125h46m19.741099892s) [noise=true , address=104.36.82.14:4222 , piece_id=D73WUVHJKYLLGRWGJEEVKEOPQYUAKJWNLLRKQAX3XPZZZDAI5QOQ, seg_pos={0 0}, upload=110885]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:54:44.388337807 (125h46m19.741122905s) [noise=true , address=104.36.82.2:4729 , piece_id=VRETJR2UUWHEGSO5N32SBCECAUYU2RGSQJCOPV7WPRLVFS6M5YRQ, seg_pos={0 0}, upload=110885]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:54:44.388385538 (125h46m19.741170636s) [seg_pos={0 0}, upload=110885]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 19:54:44.405835315 (125h46m19.758620423s) [upload=110885]: Done uploading segments... Sep 3 19:54:44.405848780 (125h46m19.758633888s) [upload=110885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 542.305256ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................X......................X..X.XXX.] duration: 486.728984ms FAIL (2025-09-03 19:55:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 486.7484ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................X......................X..X.XXX] duration: 30.001207581s FAIL (2025-09-03 19:55:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001221898s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................XX.X..X.X.] duration: 341.261576ms FAIL (2025-09-03 19:54:44 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 19:54:14.403115658 (125h45m49.755900787s) [upload=110885]: Starting upload Sep 3 19:54:14.403199486 (125h45m49.755984625s) [upload=110885]: Done starting upload Sep 3 19:54:14.403287271 (125h45m49.756072399s) [upload=110885]: Uploading segments... Sep 3 19:54:14.403649492 (125h45m49.75643462s) [upload=110885]: Got next segment. Inline: false Sep 3 19:54:14.403697302 (125h45m49.75648243s) [upload=110885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:54:14.403703914 (125h45m49.756489042s) [upload=110885]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:54:14.497238203 (125h45m49.850023331s) [seg_pos={0 0}, upload=110885]: Begin upload segment... Sep 3 19:54:14.497280222 (125h45m49.85006535s) [seg_pos={0 0}, upload=110885]: Done begin upload segment. Sep 3 19:54:14.497311981 (125h45m49.85009711s) [noise=true , address=104.36.82.16:4295 , piece_id=UMMOFFB3TF5W64MBSHSAFTNELPDTX36ZK4LQAXMXCF25GGLZBQQA, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.497817503 (125h45m49.85060263s) [noise=true , address=104.36.82.5:4838 , piece_id=KKILOVJ7AXF3O2FXGZD3COVNQWOYACCFWO3WUGPXXOQFZAS4WQ3Q, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.498182630 (125h45m49.850967757s) [noise=true , address=104.36.82.14:4222 , piece_id=D73WUVHJKYLLGRWGJEEVKEOPQYUAKJWNLLRKQAX3XPZZZDAI5QOQ, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.498550361 (125h45m49.851335489s) [noise=true , address=77.90.31.15:4698 , piece_id=PM64UCH6ICOKWW3MN4Z6J7E76VCQLF7NT4XQRPIRUZPAQ2LCVY4A, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.498942198 (125h45m49.851727336s) [noise=true , address=45.91.116.9:4262 , piece_id=NICM4KJ6KL4XFDF3M24UDPYIX2CA7VO62TOYNVDV5T5C34TMFNVA, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.499290594 (125h45m49.852075721s) [noise=true , address=45.91.116.13:4153 , piece_id=OIXRMOJZAXNBDJH4Y4ZMR2GJTPTBBMFMHUHYAHII3NZIZPLQVTYA, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.499664365 (125h45m49.852449503s) [noise=true , address=45.91.116.6:4975 , piece_id=76BSHCO66KXF7VXQAEJEAWNM7LMJY3VIOQSOVF2TIZUAMT3GZIJQ, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.499800581 (125h45m49.85258572s) [noise=true , address=45.91.116.11:4117 , piece_id=OZW5SRUEBR3HVB5OPPBLQMHFMNDSM3T4ADX5OXDUB5S3MGUWBG2Q, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.501211505 (125h45m49.853996633s) [noise=true , address=45.91.116.5:4299 , piece_id=SOQG4YIAAESZFZHQ4TPSZ6HKLOGELLDG5PCYJVN7C6CNQ5HYWJQA, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.501780004 (125h45m49.854565132s) [noise=true , address=104.36.82.2:4729 , piece_id=VRETJR2UUWHEGSO5N32SBCECAUYU2RGSQJCOPV7WPRLVFS6M5YRQ, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.502538641 (125h45m49.855323769s) [noise=true , address=77.90.31.21:4877 , piece_id=BFN2K35F33QD53MCE2C75UEDC7Q2HYEJOKVU6HYNUTRE6BKJJCFA, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.502975021 (125h45m49.85576015s) [noise=true , address=45.91.116.4:4940 , piece_id=R2NOHO4FHKJP4FRDIAB6DFRSUFTAUJJKFJW753SASSKRMKQMLX5Q, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.503470433 (125h45m49.856255562s) [noise=true , address=45.91.116.13:4154 , piece_id=4S4477Z5QVN3TMWKPILSNRBJA3OJ23YGVK36I5HDUO5IHJVSJB5A, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.503900312 (125h45m49.85668544s) [noise=true , address=45.91.116.11:4119 , piece_id=Q55DIOGQJKGXASB2OVAWPBHIY5MFL76EG5F6SIRCJPYETW4SWD3Q, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.504301726 (125h45m49.857086855s) [noise=true , address=45.91.116.6:4974 , piece_id=RZIN6NGRPAYUXXAYBTDJW6V5OVKQBJ7IQBUK5OEVIRZ53WIFRHYA, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.504666874 (125h45m49.857452001s) [noise=true , address=45.91.116.7:4011 , piece_id=YNJ4YFR67GYFCBNCO4FL5W3P2OTU4NPCZKHIIQEWIT2Y7HULPK5Q, seg_pos={0 0}, upload=110885]: Uploading piece... Sep 3 19:54:14.614557620 (125h45m49.967342747s) [upload=110885]: Next returned nil segment Sep 3 19:54:14.614573099 (125h45m49.967358227s) [upload=110885]: Waiting for error group managing segments... Sep 3 19:54:14.623997493 (125h45m49.976782621s) [noise=true , address=45.91.116.11:4119 , piece_id=Q55DIOGQJKGXASB2OVAWPBHIY5MFL76EG5F6SIRCJPYETW4SWD3Q, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.624439564 (125h45m49.977224692s) [noise=true , address=45.91.116.11:4117 , piece_id=OZW5SRUEBR3HVB5OPPBLQMHFMNDSM3T4ADX5OXDUB5S3MGUWBG2Q, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.624590588 (125h45m49.977375726s) [noise=true , address=45.91.116.13:4153 , piece_id=OIXRMOJZAXNBDJH4Y4ZMR2GJTPTBBMFMHUHYAHII3NZIZPLQVTYA, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.625540105 (125h45m49.978325232s) [noise=true , address=45.91.116.13:4154 , piece_id=4S4477Z5QVN3TMWKPILSNRBJA3OJ23YGVK36I5HDUO5IHJVSJB5A, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.633640358 (125h45m49.986425476s) [noise=true , address=45.91.116.9:4262 , piece_id=NICM4KJ6KL4XFDF3M24UDPYIX2CA7VO62TOYNVDV5T5C34TMFNVA, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.633790400 (125h45m49.986575517s) [noise=true , address=45.91.116.6:4974 , piece_id=RZIN6NGRPAYUXXAYBTDJW6V5OVKQBJ7IQBUK5OEVIRZ53WIFRHYA, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.635124730 (125h45m49.987909847s) [noise=true , address=45.91.116.4:4940 , piece_id=R2NOHO4FHKJP4FRDIAB6DFRSUFTAUJJKFJW753SASSKRMKQMLX5Q, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.635626583 (125h45m49.988411702s) [noise=true , address=45.91.116.6:4975 , piece_id=76BSHCO66KXF7VXQAEJEAWNM7LMJY3VIOQSOVF2TIZUAMT3GZIJQ, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.636979518 (125h45m49.989764626s) [noise=true , address=45.91.116.5:4299 , piece_id=SOQG4YIAAESZFZHQ4TPSZ6HKLOGELLDG5PCYJVN7C6CNQ5HYWJQA, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.641360187 (125h45m49.994145315s) [noise=true , address=45.91.116.7:4011 , piece_id=YNJ4YFR67GYFCBNCO4FL5W3P2OTU4NPCZKHIIQEWIT2Y7HULPK5Q, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.649885590 (125h45m50.002670708s) [noise=true , address=77.90.31.15:4698 , piece_id=PM64UCH6ICOKWW3MN4Z6J7E76VCQLF7NT4XQRPIRUZPAQ2LCVY4A, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:14.722385248 (125h45m50.075170376s) [noise=true , address=77.90.31.21:4877 , piece_id=BFN2K35F33QD53MCE2C75UEDC7Q2HYEJOKVU6HYNUTRE6BKJJCFA, seg_pos={0 0}, upload=110885]: Done uploading piece. err: <nil> Sep 3 19:54:44.388239984 (125h46m19.741025092s) [noise=true , address=104.36.82.5:4838 , piece_id=KKILOVJ7AXF3O2FXGZD3COVNQWOYACCFWO3WUGPXXOQFZAS4WQ3Q, seg_pos={0 0}, upload=110885]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:54:44.388282233 (125h46m19.741067341s) [noise=true , address=104.36.82.16:4295 , piece_id=UMMOFFB3TF5W64MBSHSAFTNELPDTX36ZK4LQAXMXCF25GGLZBQQA, seg_pos={0 0}, upload=110885]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:54:44.388314784 (125h46m19.741099892s) [noise=true , address=104.36.82.14:4222 , piece_id=D73WUVHJKYLLGRWGJEEVKEOPQYUAKJWNLLRKQAX3XPZZZDAI5QOQ, seg_pos={0 0}, upload=110885]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:54:44.388337807 (125h46m19.741122905s) [noise=true , address=104.36.82.2:4729 , piece_id=VRETJR2UUWHEGSO5N32SBCECAUYU2RGSQJCOPV7WPRLVFS6M5YRQ, seg_pos={0 0}, upload=110885]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:54:44.388385538 (125h46m19.741170636s) [seg_pos={0 0}, upload=110885]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 19:54:44.405835315 (125h46m19.758620423s) [upload=110885]: Done uploading segments... Sep 3 19:54:44.405848780 (125h46m19.758633888s) [upload=110885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 341.296394ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................X......................X..X.XX] duration: 30.001125404s FAIL (2025-09-03 19:54:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001140883s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................X......................X..X.X] duration: 30.000543738s FAIL (2025-09-03 19:53:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000566381s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................XX.X..X.] duration: 353.901389ms FAIL (2025-09-03 19:53:46 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 19:52:56.576663397 (125h44m31.929448495s) [upload=110865]: Starting upload Sep 3 19:52:56.576710896 (125h44m31.929495994s) [upload=110865]: Done starting upload Sep 3 19:52:56.576926732 (125h44m31.92971184s) [upload=110865]: Uploading segments... Sep 3 19:52:56.576950646 (125h44m31.929735745s) [upload=110865]: Got next segment. Inline: false Sep 3 19:52:56.576975132 (125h44m31.92976024s) [upload=110865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:52:56.576978579 (125h44m31.929763687s) [upload=110865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:52:56.600712774 (125h44m31.953497903s) [seg_pos={0 0}, upload=110865]: Begin upload segment... Sep 3 19:52:56.600767127 (125h44m31.953552254s) [seg_pos={0 0}, upload=110865]: Done begin upload segment. Sep 3 19:52:56.600801982 (125h44m31.953587111s) [noise=true , address=45.91.116.7:4011 , piece_id=HFJMTM5UE3MHBHUA7SYK5WKCTYRUD7N2MUSU7XU5U4E44444U6XA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.600840325 (125h44m31.953625423s) [noise=true , address=45.91.116.9:4261 , piece_id=HIR7PR5CJTMT35OZ2ZGXU4MOPDHRH4G2IIHMTYOFCYMECUOYSLCQ, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.601221351 (125h44m31.95400645s) [noise=true , address=45.91.116.13:4154 , piece_id=RZMKKATF22DDU4UOXTCOQK4OJERDYYKAUSS6ITJM6WTQRLLFHAXA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.601244174 (125h44m31.954029302s) [noise=true , address=45.91.116.5:4298 , piece_id=CNZH6KM5HHD327KJBDRUQJSU63YTXIORPPRV2PPJAB3AM5LBO52A, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.601487702 (125h44m31.95427281s) [noise=true , address=45.91.116.10:4083 , piece_id=A6SL3T6IJRKL3F4KOB4TJJO4MSB7S36V3P4ZVF3OH45VZ7HILEOA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.601618428 (125h44m31.954403556s) [noise=true , address=104.36.82.3:4767 , piece_id=XS62MUUROSUDZMLYZ4M2NUECABH3NLFTMU3MK7UGHQZT4F6I4F6Q, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.601823453 (125h44m31.954608562s) [noise=true , address=45.91.116.6:4973 , piece_id=ULXW4YBLELNTIAIEXK76AZAFFCD6DSNZZ6UMDV6D65KCVUQGYVMA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.602041633 (125h44m31.954826762s) [noise=true , address=45.91.116.8:4048 , piece_id=ZKWY5AJOOUTJ5UVRKZIPF24RJW6LZKL3LDRWU2NENGH2HA7NPEBA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.602066781 (125h44m31.954851889s) [noise=true , address=77.90.31.19:4842 , piece_id=NBT5LB6HZ6JGEU3NRNZQXNXT56C2HX5QJRBNTNAH6C45XWW2HKNA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.602330037 (125h44m31.955115145s) [noise=true , address=104.36.82.4:4801 , piece_id=JIX6L57BEBNRE7RD5GOGY4RFVFIZKXZKVOBCIWMG443XIXSPPN4A, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.602394628 (125h44m31.955179756s) [noise=true , address=104.36.82.7:4910 , piece_id=PRCG6L5HO7VKE2XE373BNGPYCQ2PCJH5WGJCIS2Z6VF5OMNEFYAQ, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.602582851 (125h44m31.95536795s) [noise=true , address=77.90.31.18:4805 , piece_id=XJZKV3F4AICQMSEJN5UCODYYARC4RCWSXQQGJBUEQNEBMK572C5A, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.602779291 (125h44m31.95556442s) [noise=true , address=45.91.116.6:4974 , piece_id=6ON4ZUXVUVKQDISNFXB5RC63ES2XEXWIIO4CT6E3ICHD5P2MI6CA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.602830808 (125h44m31.955615916s) [noise=true , address=77.90.31.13:4627 , piece_id=R3Z2V56QLOL6DY6JGRJD52GKXUFZXHPYCWI63TBN2WSI5LMDCXXA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.603117307 (125h44m31.955902425s) [noise=true , address=45.91.116.8:4047 , piece_id=K6Z5HFPVTOAFX2WM3JGHCVMDMFX6LVPPE5SQBQA375Q7BDNX6KOA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.603552826 (125h44m31.956337954s) [noise=true , address=104.36.82.5:4837 , piece_id=JCVOKZWK6IAXAHXEMAF7VZL7ZNCS2M5XMXPCFUGF4W7FMYESTY4Q, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.687449126 (125h44m32.040234234s) [upload=110865]: Next returned nil segment Sep 3 19:52:56.687463263 (125h44m32.040248371s) [upload=110865]: Waiting for error group managing segments... Sep 3 19:52:56.698047639 (125h44m32.050832757s) [noise=true , address=45.91.116.13:4154 , piece_id=RZMKKATF22DDU4UOXTCOQK4OJERDYYKAUSS6ITJM6WTQRLLFHAXA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.698813419 (125h44m32.051598527s) [noise=true , address=45.91.116.9:4261 , piece_id=HIR7PR5CJTMT35OZ2ZGXU4MOPDHRH4G2IIHMTYOFCYMECUOYSLCQ, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.706766316 (125h44m32.059551434s) [noise=true , address=45.91.116.8:4048 , piece_id=ZKWY5AJOOUTJ5UVRKZIPF24RJW6LZKL3LDRWU2NENGH2HA7NPEBA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.710787599 (125h44m32.063572697s) [noise=true , address=77.90.31.19:4842 , piece_id=NBT5LB6HZ6JGEU3NRNZQXNXT56C2HX5QJRBNTNAH6C45XWW2HKNA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.720048285 (125h44m32.072833413s) [noise=true , address=45.91.116.6:4974 , piece_id=6ON4ZUXVUVKQDISNFXB5RC63ES2XEXWIIO4CT6E3ICHD5P2MI6CA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.724046484 (125h44m32.076831612s) [noise=true , address=45.91.116.10:4083 , piece_id=A6SL3T6IJRKL3F4KOB4TJJO4MSB7S36V3P4ZVF3OH45VZ7HILEOA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.729118053 (125h44m32.081903151s) [noise=true , address=45.91.116.8:4047 , piece_id=K6Z5HFPVTOAFX2WM3JGHCVMDMFX6LVPPE5SQBQA375Q7BDNX6KOA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.733077579 (125h44m32.085862707s) [noise=true , address=45.91.116.7:4011 , piece_id=HFJMTM5UE3MHBHUA7SYK5WKCTYRUD7N2MUSU7XU5U4E44444U6XA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.765288216 (125h44m32.118073344s) [noise=true , address=77.90.31.13:4627 , piece_id=R3Z2V56QLOL6DY6JGRJD52GKXUFZXHPYCWI63TBN2WSI5LMDCXXA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.793482930 (125h44m32.146268028s) [noise=true , address=77.90.31.18:4805 , piece_id=XJZKV3F4AICQMSEJN5UCODYYARC4RCWSXQQGJBUEQNEBMK572C5A, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:57.070009615 (125h44m32.422794723s) [noise=true , address=45.91.116.5:4298 , piece_id=CNZH6KM5HHD327KJBDRUQJSU63YTXIORPPRV2PPJAB3AM5LBO52A, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:57.160893337 (125h44m32.513678445s) [noise=true , address=45.91.116.6:4973 , piece_id=ULXW4YBLELNTIAIEXK76AZAFFCD6DSNZZ6UMDV6D65KCVUQGYVMA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:53:26.564392900 (125h45m1.917178008s) [noise=true , address=104.36.82.3:4767 , piece_id=XS62MUUROSUDZMLYZ4M2NUECABH3NLFTMU3MK7UGHQZT4F6I4F6Q, seg_pos={0 0}, upload=110865]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 5.161.89.252:34086->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:53:26.564428407 (125h45m1.917213515s) [noise=true , address=104.36.82.4:4801 , piece_id=JIX6L57BEBNRE7RD5GOGY4RFVFIZKXZKVOBCIWMG443XIXSPPN4A, seg_pos={0 0}, upload=110865]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 5.161.89.252:23592->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:53:26.564439117 (125h45m1.917224215s) [noise=true , address=104.36.82.7:4910 , piece_id=PRCG6L5HO7VKE2XE373BNGPYCQ2PCJH5WGJCIS2Z6VF5OMNEFYAQ, seg_pos={0 0}, upload=110865]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 5.161.89.252:52472->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:53:26.564443485 (125h45m1.917228593s) [noise=true , address=104.36.82.5:4837 , piece_id=JCVOKZWK6IAXAHXEMAF7VZL7ZNCS2M5XMXPCFUGF4W7FMYESTY4Q, seg_pos={0 0}, upload=110865]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:53:26.564462361 (125h45m1.917247469s) [seg_pos={0 0}, upload=110865]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 19:53:26.577174377 (125h45m1.929959485s) [upload=110865]: Done uploading segments... Sep 3 19:53:26.577186691 (125h45m1.929971789s) [upload=110865]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 353.922709ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................XX.X..X] duration: 30.014384462s FAIL (2025-09-03 19:53:26 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 19:52:56.576663397 (125h44m31.929448495s) [upload=110865]: Starting upload Sep 3 19:52:56.576710896 (125h44m31.929495994s) [upload=110865]: Done starting upload Sep 3 19:52:56.576926732 (125h44m31.92971184s) [upload=110865]: Uploading segments... Sep 3 19:52:56.576950646 (125h44m31.929735745s) [upload=110865]: Got next segment. Inline: false Sep 3 19:52:56.576975132 (125h44m31.92976024s) [upload=110865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:52:56.576978579 (125h44m31.929763687s) [upload=110865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:52:56.600712774 (125h44m31.953497903s) [seg_pos={0 0}, upload=110865]: Begin upload segment... Sep 3 19:52:56.600767127 (125h44m31.953552254s) [seg_pos={0 0}, upload=110865]: Done begin upload segment. Sep 3 19:52:56.600801982 (125h44m31.953587111s) [noise=true , address=45.91.116.7:4011 , piece_id=HFJMTM5UE3MHBHUA7SYK5WKCTYRUD7N2MUSU7XU5U4E44444U6XA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.600840325 (125h44m31.953625423s) [noise=true , address=45.91.116.9:4261 , piece_id=HIR7PR5CJTMT35OZ2ZGXU4MOPDHRH4G2IIHMTYOFCYMECUOYSLCQ, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.601221351 (125h44m31.95400645s) [noise=true , address=45.91.116.13:4154 , piece_id=RZMKKATF22DDU4UOXTCOQK4OJERDYYKAUSS6ITJM6WTQRLLFHAXA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.601244174 (125h44m31.954029302s) [noise=true , address=45.91.116.5:4298 , piece_id=CNZH6KM5HHD327KJBDRUQJSU63YTXIORPPRV2PPJAB3AM5LBO52A, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.601487702 (125h44m31.95427281s) [noise=true , address=45.91.116.10:4083 , piece_id=A6SL3T6IJRKL3F4KOB4TJJO4MSB7S36V3P4ZVF3OH45VZ7HILEOA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.601618428 (125h44m31.954403556s) [noise=true , address=104.36.82.3:4767 , piece_id=XS62MUUROSUDZMLYZ4M2NUECABH3NLFTMU3MK7UGHQZT4F6I4F6Q, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.601823453 (125h44m31.954608562s) [noise=true , address=45.91.116.6:4973 , piece_id=ULXW4YBLELNTIAIEXK76AZAFFCD6DSNZZ6UMDV6D65KCVUQGYVMA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.602041633 (125h44m31.954826762s) [noise=true , address=45.91.116.8:4048 , piece_id=ZKWY5AJOOUTJ5UVRKZIPF24RJW6LZKL3LDRWU2NENGH2HA7NPEBA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.602066781 (125h44m31.954851889s) [noise=true , address=77.90.31.19:4842 , piece_id=NBT5LB6HZ6JGEU3NRNZQXNXT56C2HX5QJRBNTNAH6C45XWW2HKNA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.602330037 (125h44m31.955115145s) [noise=true , address=104.36.82.4:4801 , piece_id=JIX6L57BEBNRE7RD5GOGY4RFVFIZKXZKVOBCIWMG443XIXSPPN4A, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.602394628 (125h44m31.955179756s) [noise=true , address=104.36.82.7:4910 , piece_id=PRCG6L5HO7VKE2XE373BNGPYCQ2PCJH5WGJCIS2Z6VF5OMNEFYAQ, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.602582851 (125h44m31.95536795s) [noise=true , address=77.90.31.18:4805 , piece_id=XJZKV3F4AICQMSEJN5UCODYYARC4RCWSXQQGJBUEQNEBMK572C5A, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.602779291 (125h44m31.95556442s) [noise=true , address=45.91.116.6:4974 , piece_id=6ON4ZUXVUVKQDISNFXB5RC63ES2XEXWIIO4CT6E3ICHD5P2MI6CA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.602830808 (125h44m31.955615916s) [noise=true , address=77.90.31.13:4627 , piece_id=R3Z2V56QLOL6DY6JGRJD52GKXUFZXHPYCWI63TBN2WSI5LMDCXXA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.603117307 (125h44m31.955902425s) [noise=true , address=45.91.116.8:4047 , piece_id=K6Z5HFPVTOAFX2WM3JGHCVMDMFX6LVPPE5SQBQA375Q7BDNX6KOA, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.603552826 (125h44m31.956337954s) [noise=true , address=104.36.82.5:4837 , piece_id=JCVOKZWK6IAXAHXEMAF7VZL7ZNCS2M5XMXPCFUGF4W7FMYESTY4Q, seg_pos={0 0}, upload=110865]: Uploading piece... Sep 3 19:52:56.687449126 (125h44m32.040234234s) [upload=110865]: Next returned nil segment Sep 3 19:52:56.687463263 (125h44m32.040248371s) [upload=110865]: Waiting for error group managing segments... Sep 3 19:52:56.698047639 (125h44m32.050832757s) [noise=true , address=45.91.116.13:4154 , piece_id=RZMKKATF22DDU4UOXTCOQK4OJERDYYKAUSS6ITJM6WTQRLLFHAXA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.698813419 (125h44m32.051598527s) [noise=true , address=45.91.116.9:4261 , piece_id=HIR7PR5CJTMT35OZ2ZGXU4MOPDHRH4G2IIHMTYOFCYMECUOYSLCQ, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.706766316 (125h44m32.059551434s) [noise=true , address=45.91.116.8:4048 , piece_id=ZKWY5AJOOUTJ5UVRKZIPF24RJW6LZKL3LDRWU2NENGH2HA7NPEBA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.710787599 (125h44m32.063572697s) [noise=true , address=77.90.31.19:4842 , piece_id=NBT5LB6HZ6JGEU3NRNZQXNXT56C2HX5QJRBNTNAH6C45XWW2HKNA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.720048285 (125h44m32.072833413s) [noise=true , address=45.91.116.6:4974 , piece_id=6ON4ZUXVUVKQDISNFXB5RC63ES2XEXWIIO4CT6E3ICHD5P2MI6CA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.724046484 (125h44m32.076831612s) [noise=true , address=45.91.116.10:4083 , piece_id=A6SL3T6IJRKL3F4KOB4TJJO4MSB7S36V3P4ZVF3OH45VZ7HILEOA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.729118053 (125h44m32.081903151s) [noise=true , address=45.91.116.8:4047 , piece_id=K6Z5HFPVTOAFX2WM3JGHCVMDMFX6LVPPE5SQBQA375Q7BDNX6KOA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.733077579 (125h44m32.085862707s) [noise=true , address=45.91.116.7:4011 , piece_id=HFJMTM5UE3MHBHUA7SYK5WKCTYRUD7N2MUSU7XU5U4E44444U6XA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.765288216 (125h44m32.118073344s) [noise=true , address=77.90.31.13:4627 , piece_id=R3Z2V56QLOL6DY6JGRJD52GKXUFZXHPYCWI63TBN2WSI5LMDCXXA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:56.793482930 (125h44m32.146268028s) [noise=true , address=77.90.31.18:4805 , piece_id=XJZKV3F4AICQMSEJN5UCODYYARC4RCWSXQQGJBUEQNEBMK572C5A, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:57.070009615 (125h44m32.422794723s) [noise=true , address=45.91.116.5:4298 , piece_id=CNZH6KM5HHD327KJBDRUQJSU63YTXIORPPRV2PPJAB3AM5LBO52A, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:52:57.160893337 (125h44m32.513678445s) [noise=true , address=45.91.116.6:4973 , piece_id=ULXW4YBLELNTIAIEXK76AZAFFCD6DSNZZ6UMDV6D65KCVUQGYVMA, seg_pos={0 0}, upload=110865]: Done uploading piece. err: <nil> Sep 3 19:53:26.564392900 (125h45m1.917178008s) [noise=true , address=104.36.82.3:4767 , piece_id=XS62MUUROSUDZMLYZ4M2NUECABH3NLFTMU3MK7UGHQZT4F6I4F6Q, seg_pos={0 0}, upload=110865]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 5.161.89.252:34086->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:53:26.564428407 (125h45m1.917213515s) [noise=true , address=104.36.82.4:4801 , piece_id=JIX6L57BEBNRE7RD5GOGY4RFVFIZKXZKVOBCIWMG443XIXSPPN4A, seg_pos={0 0}, upload=110865]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 5.161.89.252:23592->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:53:26.564439117 (125h45m1.917224215s) [noise=true , address=104.36.82.7:4910 , piece_id=PRCG6L5HO7VKE2XE373BNGPYCQ2PCJH5WGJCIS2Z6VF5OMNEFYAQ, seg_pos={0 0}, upload=110865]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 5.161.89.252:52472->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:53:26.564443485 (125h45m1.917228593s) [noise=true , address=104.36.82.5:4837 , piece_id=JCVOKZWK6IAXAHXEMAF7VZL7ZNCS2M5XMXPCFUGF4W7FMYESTY4Q, seg_pos={0 0}, upload=110865]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:53:26.564462361 (125h45m1.917247469s) [seg_pos={0 0}, upload=110865]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 19:53:26.577174377 (125h45m1.929959485s) [upload=110865]: Done uploading segments... Sep 3 19:53:26.577186691 (125h45m1.929971789s) [upload=110865]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.014413026s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X......................X..X.] duration: 482.168446ms FAIL (2025-09-03 19:53:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 482.227889ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X......................X..X] duration: 30.000669891s FAIL (2025-09-03 19:52:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000685871s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................XX.X..] duration: 496.168306ms FAIL (2025-09-03 19:52:17 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 3 19:50:28.574777387 (125h42m3.927562515s) [upload=110833]: Starting upload Sep 3 19:50:28.574845545 (125h42m3.927630703s) [upload=110833]: Done starting upload Sep 3 19:50:28.575158644 (125h42m3.927943773s) [upload=110833]: Uploading segments... Sep 3 19:50:28.575255466 (125h42m3.928040604s) [upload=110833]: Got next segment. Inline: false Sep 3 19:50:28.575296303 (125h42m3.928081432s) [upload=110833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:50:28.575303186 (125h42m3.928088325s) [upload=110833]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:50:28.600159382 (125h42m3.95294451s) [seg_pos={0 0}, upload=110833]: Begin upload segment... Sep 3 19:50:28.600197233 (125h42m3.952982361s) [seg_pos={0 0}, upload=110833]: Done begin upload segment. Sep 3 19:50:28.600228542 (125h42m3.95301367s) [noise=true , address=77.90.31.16:4733 , piece_id=HOFKXNJBOJ27QNAWARHZSRA2F7R4XN2VAGY4TRLQ4BLQ6SZNBEMA, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.600255563 (125h42m3.953040721s) [noise=true , address=45.91.116.12:4225 , piece_id=5OV6BI6KAO3KP5URFJYGRBQHY3XFNJNY52DMB5O3RBSL4XU4KS4Q, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.600347345 (125h42m3.953132474s) [noise=true , address=45.91.116.9:4261 , piece_id=67JORI3KBKETLOSVQQWDVBBYPN7LSKAQQOXUMUWATHUGBIMCWA6Q, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.600770010 (125h42m3.953555139s) [noise=true , address=45.91.116.6:4975 , piece_id=QSNIWRITJ2UKMDYUJOJG3OJAJW2O4X76R6G5HCZ3BA4UZWVHBGEQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.600815676 (125h42m3.953601766s) [noise=true , address=45.91.116.4:4937 , piece_id=VFILXM44M7EU7XZWZTBEETATMMFA7IBXAAXYZFBIV3TBULYRFEWQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.601267617 (125h42m3.954052744s) [noise=true , address=104.36.82.6:4875 , piece_id=K3MQHDJ7BQ7QKBFY26U6XXNHMHMITUAW73SENK5XZCCGP5BMGCYQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.601338540 (125h42m3.954123668s) [noise=true , address=104.36.82.9:4007 , piece_id=AEGR6QK3SJTGSHHPVSJHTFJXNANSGNUA2ZIWEG3XHE5XSHQU2SQA, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.601701652 (125h42m3.954486791s) [noise=true , address=104.36.82.13:4187 , piece_id=4WYLRG25O5MREEZAI4RPTOTTTCDQLWUX7PRQGBRKOAZADW3AL5AA, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.601765142 (125h42m3.95455027s) [noise=true , address=104.36.82.7:4912 , piece_id=FS2EN4BM234GJL4YDODWRZVL77Z3BBSJFUQCFHLEHR224FMKLC2A, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.602663972 (125h42m3.955449101s) [noise=true , address=77.90.31.17:4770 , piece_id=RGYLLEBJSTESSSVA7UZD6V4FD32OS3I2Q7ITF23LWCSR34HWP5WQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.602815948 (125h42m3.955601086s) [noise=true , address=45.91.116.5:4297 , piece_id=VCD2L2LRDRDL3RL35XO3S5HAHLDESR2DY5F2V3SDY3PPDO3J63GA, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.603514982 (125h42m3.956300141s) [noise=true , address=45.91.116.8:4047 , piece_id=SMRGHSUUE5IE7PU4UWOOZUS5U6LFOGJNEWB4IUAJVESLGDYOSVTQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.603574185 (125h42m3.956359313s) [noise=true , address=45.91.116.13:4154 , piece_id=ISYL6B7DUA4ODXTRRVITR2BCXUVQGAC4LFDKBQSYF5AHX2HYHBCA, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.604871624 (125h42m3.957656762s) [noise=true , address=45.91.116.6:4974 , piece_id=HRFFQ22SADG7PTKGCNFIQ3VSKCBHG3JJDX6VLIR5MSR3T7ZHTP2A, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.605228004 (125h42m3.958013142s) [noise=true , address=104.36.82.10:4077 , piece_id=LZCEBG5HO3IJWP3GPIATPLXXXDMOQVXHMHKT2HVFWHGJU7E42FNQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.606132837 (125h42m3.958917964s) [noise=true , address=45.91.116.10:4083 , piece_id=5AMQUYP7LMRVG22OENG66EV53526GX4DULTG3OQTNR2STDTYID6A, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.734930601 (125h42m4.087715719s) [upload=110833]: Next returned nil segment Sep 3 19:50:28.734943967 (125h42m4.087729075s) [upload=110833]: Waiting for error group managing segments... Sep 3 19:50:28.743859193 (125h42m4.096644311s) [noise=true , address=45.91.116.6:4974 , piece_id=HRFFQ22SADG7PTKGCNFIQ3VSKCBHG3JJDX6VLIR5MSR3T7ZHTP2A, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.744038891 (125h42m4.09682402s) [noise=true , address=45.91.116.5:4297 , piece_id=VCD2L2LRDRDL3RL35XO3S5HAHLDESR2DY5F2V3SDY3PPDO3J63GA, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.744641605 (125h42m4.097426713s) [noise=true , address=45.91.116.6:4975 , piece_id=QSNIWRITJ2UKMDYUJOJG3OJAJW2O4X76R6G5HCZ3BA4UZWVHBGEQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.744758946 (125h42m4.097544074s) [noise=true , address=77.90.31.17:4770 , piece_id=RGYLLEBJSTESSSVA7UZD6V4FD32OS3I2Q7ITF23LWCSR34HWP5WQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.745191980 (125h42m4.097977098s) [noise=true , address=45.91.116.12:4225 , piece_id=5OV6BI6KAO3KP5URFJYGRBQHY3XFNJNY52DMB5O3RBSL4XU4KS4Q, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.745786588 (125h42m4.098571716s) [noise=true , address=45.91.116.10:4083 , piece_id=5AMQUYP7LMRVG22OENG66EV53526GX4DULTG3OQTNR2STDTYID6A, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.746582645 (125h42m4.099367763s) [noise=true , address=45.91.116.4:4937 , piece_id=VFILXM44M7EU7XZWZTBEETATMMFA7IBXAAXYZFBIV3TBULYRFEWQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.756988735 (125h42m4.109773854s) [noise=true , address=45.91.116.8:4047 , piece_id=SMRGHSUUE5IE7PU4UWOOZUS5U6LFOGJNEWB4IUAJVESLGDYOSVTQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.759436769 (125h42m4.112221897s) [noise=true , address=45.91.116.9:4261 , piece_id=67JORI3KBKETLOSVQQWDVBBYPN7LSKAQQOXUMUWATHUGBIMCWA6Q, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.775962409 (125h42m4.128747547s) [noise=true , address=45.91.116.13:4154 , piece_id=ISYL6B7DUA4ODXTRRVITR2BCXUVQGAC4LFDKBQSYF5AHX2HYHBCA, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.896587655 (125h42m4.249372794s) [noise=true , address=77.90.31.16:4733 , piece_id=HOFKXNJBOJ27QNAWARHZSRA2F7R4XN2VAGY4TRLQ4BLQ6SZNBEMA, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:58.557541379 (125h42m33.910326487s) [noise=true , address=104.36.82.9:4007 , piece_id=AEGR6QK3SJTGSHHPVSJHTFJXNANSGNUA2ZIWEG3XHE5XSHQU2SQA, seg_pos={0 0}, upload=110833]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: context deadline exceeded; context deadline exceeded Sep 3 19:50:58.557639383 (125h42m33.910424491s) [noise=true , address=104.36.82.7:4912 , piece_id=FS2EN4BM234GJL4YDODWRZVL77Z3BBSJFUQCFHLEHR224FMKLC2A, seg_pos={0 0}, upload=110833]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 5.161.89.252:8574->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:50:58.557659522 (125h42m33.91044463s) [noise=true , address=104.36.82.13:4187 , piece_id=4WYLRG25O5MREEZAI4RPTOTTTCDQLWUX7PRQGBRKOAZADW3AL5AA, seg_pos={0 0}, upload=110833]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Sep 3 19:50:58.557734222 (125h42m33.91051933s) [noise=true , address=104.36.82.6:4875 , piece_id=K3MQHDJ7BQ7QKBFY26U6XXNHMHMITUAW73SENK5XZCCGP5BMGCYQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:50:58.557832475 (125h42m33.910617584s) [noise=true , address=104.36.82.10:4077 , piece_id=LZCEBG5HO3IJWP3GPIATPLXXXDMOQVXHMHKT2HVFWHGJU7E42FNQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:50:58.557867893 (125h42m33.910653001s) [seg_pos={0 0}, upload=110833]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 3 19:50:58.572363874 (125h42m33.925148982s) [upload=110833]: Done uploading segments... Sep 3 19:50:58.572377410 (125h42m33.925162508s) [upload=110833]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 496.195467ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X......................X..] duration: 420.825431ms FAIL (2025-09-03 19:51:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 420.860408ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XX.X.] duration: 340.724049ms FAIL (2025-09-03 19:51:24 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 3 19:50:28.574777387 (125h42m3.927562515s) [upload=110833]: Starting upload Sep 3 19:50:28.574845545 (125h42m3.927630703s) [upload=110833]: Done starting upload Sep 3 19:50:28.575158644 (125h42m3.927943773s) [upload=110833]: Uploading segments... Sep 3 19:50:28.575255466 (125h42m3.928040604s) [upload=110833]: Got next segment. Inline: false Sep 3 19:50:28.575296303 (125h42m3.928081432s) [upload=110833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:50:28.575303186 (125h42m3.928088325s) [upload=110833]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:50:28.600159382 (125h42m3.95294451s) [seg_pos={0 0}, upload=110833]: Begin upload segment... Sep 3 19:50:28.600197233 (125h42m3.952982361s) [seg_pos={0 0}, upload=110833]: Done begin upload segment. Sep 3 19:50:28.600228542 (125h42m3.95301367s) [noise=true , address=77.90.31.16:4733 , piece_id=HOFKXNJBOJ27QNAWARHZSRA2F7R4XN2VAGY4TRLQ4BLQ6SZNBEMA, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.600255563 (125h42m3.953040721s) [noise=true , address=45.91.116.12:4225 , piece_id=5OV6BI6KAO3KP5URFJYGRBQHY3XFNJNY52DMB5O3RBSL4XU4KS4Q, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.600347345 (125h42m3.953132474s) [noise=true , address=45.91.116.9:4261 , piece_id=67JORI3KBKETLOSVQQWDVBBYPN7LSKAQQOXUMUWATHUGBIMCWA6Q, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.600770010 (125h42m3.953555139s) [noise=true , address=45.91.116.6:4975 , piece_id=QSNIWRITJ2UKMDYUJOJG3OJAJW2O4X76R6G5HCZ3BA4UZWVHBGEQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.600815676 (125h42m3.953601766s) [noise=true , address=45.91.116.4:4937 , piece_id=VFILXM44M7EU7XZWZTBEETATMMFA7IBXAAXYZFBIV3TBULYRFEWQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.601267617 (125h42m3.954052744s) [noise=true , address=104.36.82.6:4875 , piece_id=K3MQHDJ7BQ7QKBFY26U6XXNHMHMITUAW73SENK5XZCCGP5BMGCYQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.601338540 (125h42m3.954123668s) [noise=true , address=104.36.82.9:4007 , piece_id=AEGR6QK3SJTGSHHPVSJHTFJXNANSGNUA2ZIWEG3XHE5XSHQU2SQA, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.601701652 (125h42m3.954486791s) [noise=true , address=104.36.82.13:4187 , piece_id=4WYLRG25O5MREEZAI4RPTOTTTCDQLWUX7PRQGBRKOAZADW3AL5AA, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.601765142 (125h42m3.95455027s) [noise=true , address=104.36.82.7:4912 , piece_id=FS2EN4BM234GJL4YDODWRZVL77Z3BBSJFUQCFHLEHR224FMKLC2A, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.602663972 (125h42m3.955449101s) [noise=true , address=77.90.31.17:4770 , piece_id=RGYLLEBJSTESSSVA7UZD6V4FD32OS3I2Q7ITF23LWCSR34HWP5WQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.602815948 (125h42m3.955601086s) [noise=true , address=45.91.116.5:4297 , piece_id=VCD2L2LRDRDL3RL35XO3S5HAHLDESR2DY5F2V3SDY3PPDO3J63GA, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.603514982 (125h42m3.956300141s) [noise=true , address=45.91.116.8:4047 , piece_id=SMRGHSUUE5IE7PU4UWOOZUS5U6LFOGJNEWB4IUAJVESLGDYOSVTQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.603574185 (125h42m3.956359313s) [noise=true , address=45.91.116.13:4154 , piece_id=ISYL6B7DUA4ODXTRRVITR2BCXUVQGAC4LFDKBQSYF5AHX2HYHBCA, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.604871624 (125h42m3.957656762s) [noise=true , address=45.91.116.6:4974 , piece_id=HRFFQ22SADG7PTKGCNFIQ3VSKCBHG3JJDX6VLIR5MSR3T7ZHTP2A, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.605228004 (125h42m3.958013142s) [noise=true , address=104.36.82.10:4077 , piece_id=LZCEBG5HO3IJWP3GPIATPLXXXDMOQVXHMHKT2HVFWHGJU7E42FNQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.606132837 (125h42m3.958917964s) [noise=true , address=45.91.116.10:4083 , piece_id=5AMQUYP7LMRVG22OENG66EV53526GX4DULTG3OQTNR2STDTYID6A, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.734930601 (125h42m4.087715719s) [upload=110833]: Next returned nil segment Sep 3 19:50:28.734943967 (125h42m4.087729075s) [upload=110833]: Waiting for error group managing segments... Sep 3 19:50:28.743859193 (125h42m4.096644311s) [noise=true , address=45.91.116.6:4974 , piece_id=HRFFQ22SADG7PTKGCNFIQ3VSKCBHG3JJDX6VLIR5MSR3T7ZHTP2A, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.744038891 (125h42m4.09682402s) [noise=true , address=45.91.116.5:4297 , piece_id=VCD2L2LRDRDL3RL35XO3S5HAHLDESR2DY5F2V3SDY3PPDO3J63GA, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.744641605 (125h42m4.097426713s) [noise=true , address=45.91.116.6:4975 , piece_id=QSNIWRITJ2UKMDYUJOJG3OJAJW2O4X76R6G5HCZ3BA4UZWVHBGEQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.744758946 (125h42m4.097544074s) [noise=true , address=77.90.31.17:4770 , piece_id=RGYLLEBJSTESSSVA7UZD6V4FD32OS3I2Q7ITF23LWCSR34HWP5WQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.745191980 (125h42m4.097977098s) [noise=true , address=45.91.116.12:4225 , piece_id=5OV6BI6KAO3KP5URFJYGRBQHY3XFNJNY52DMB5O3RBSL4XU4KS4Q, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.745786588 (125h42m4.098571716s) [noise=true , address=45.91.116.10:4083 , piece_id=5AMQUYP7LMRVG22OENG66EV53526GX4DULTG3OQTNR2STDTYID6A, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.746582645 (125h42m4.099367763s) [noise=true , address=45.91.116.4:4937 , piece_id=VFILXM44M7EU7XZWZTBEETATMMFA7IBXAAXYZFBIV3TBULYRFEWQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.756988735 (125h42m4.109773854s) [noise=true , address=45.91.116.8:4047 , piece_id=SMRGHSUUE5IE7PU4UWOOZUS5U6LFOGJNEWB4IUAJVESLGDYOSVTQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.759436769 (125h42m4.112221897s) [noise=true , address=45.91.116.9:4261 , piece_id=67JORI3KBKETLOSVQQWDVBBYPN7LSKAQQOXUMUWATHUGBIMCWA6Q, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.775962409 (125h42m4.128747547s) [noise=true , address=45.91.116.13:4154 , piece_id=ISYL6B7DUA4ODXTRRVITR2BCXUVQGAC4LFDKBQSYF5AHX2HYHBCA, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.896587655 (125h42m4.249372794s) [noise=true , address=77.90.31.16:4733 , piece_id=HOFKXNJBOJ27QNAWARHZSRA2F7R4XN2VAGY4TRLQ4BLQ6SZNBEMA, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:58.557541379 (125h42m33.910326487s) [noise=true , address=104.36.82.9:4007 , piece_id=AEGR6QK3SJTGSHHPVSJHTFJXNANSGNUA2ZIWEG3XHE5XSHQU2SQA, seg_pos={0 0}, upload=110833]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: context deadline exceeded; context deadline exceeded Sep 3 19:50:58.557639383 (125h42m33.910424491s) [noise=true , address=104.36.82.7:4912 , piece_id=FS2EN4BM234GJL4YDODWRZVL77Z3BBSJFUQCFHLEHR224FMKLC2A, seg_pos={0 0}, upload=110833]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 5.161.89.252:8574->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:50:58.557659522 (125h42m33.91044463s) [noise=true , address=104.36.82.13:4187 , piece_id=4WYLRG25O5MREEZAI4RPTOTTTCDQLWUX7PRQGBRKOAZADW3AL5AA, seg_pos={0 0}, upload=110833]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Sep 3 19:50:58.557734222 (125h42m33.91051933s) [noise=true , address=104.36.82.6:4875 , piece_id=K3MQHDJ7BQ7QKBFY26U6XXNHMHMITUAW73SENK5XZCCGP5BMGCYQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:50:58.557832475 (125h42m33.910617584s) [noise=true , address=104.36.82.10:4077 , piece_id=LZCEBG5HO3IJWP3GPIATPLXXXDMOQVXHMHKT2HVFWHGJU7E42FNQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:50:58.557867893 (125h42m33.910653001s) [seg_pos={0 0}, upload=110833]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 3 19:50:58.572363874 (125h42m33.925148982s) [upload=110833]: Done uploading segments... Sep 3 19:50:58.572377410 (125h42m33.925162508s) [upload=110833]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 340.746451ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XX.X] duration: 30.016954242s FAIL (2025-09-03 19:50:58 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 3 19:50:28.574777387 (125h42m3.927562515s) [upload=110833]: Starting upload Sep 3 19:50:28.574845545 (125h42m3.927630703s) [upload=110833]: Done starting upload Sep 3 19:50:28.575158644 (125h42m3.927943773s) [upload=110833]: Uploading segments... Sep 3 19:50:28.575255466 (125h42m3.928040604s) [upload=110833]: Got next segment. Inline: false Sep 3 19:50:28.575296303 (125h42m3.928081432s) [upload=110833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:50:28.575303186 (125h42m3.928088325s) [upload=110833]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:50:28.600159382 (125h42m3.95294451s) [seg_pos={0 0}, upload=110833]: Begin upload segment... Sep 3 19:50:28.600197233 (125h42m3.952982361s) [seg_pos={0 0}, upload=110833]: Done begin upload segment. Sep 3 19:50:28.600228542 (125h42m3.95301367s) [noise=true , address=77.90.31.16:4733 , piece_id=HOFKXNJBOJ27QNAWARHZSRA2F7R4XN2VAGY4TRLQ4BLQ6SZNBEMA, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.600255563 (125h42m3.953040721s) [noise=true , address=45.91.116.12:4225 , piece_id=5OV6BI6KAO3KP5URFJYGRBQHY3XFNJNY52DMB5O3RBSL4XU4KS4Q, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.600347345 (125h42m3.953132474s) [noise=true , address=45.91.116.9:4261 , piece_id=67JORI3KBKETLOSVQQWDVBBYPN7LSKAQQOXUMUWATHUGBIMCWA6Q, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.600770010 (125h42m3.953555139s) [noise=true , address=45.91.116.6:4975 , piece_id=QSNIWRITJ2UKMDYUJOJG3OJAJW2O4X76R6G5HCZ3BA4UZWVHBGEQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.600815676 (125h42m3.953601766s) [noise=true , address=45.91.116.4:4937 , piece_id=VFILXM44M7EU7XZWZTBEETATMMFA7IBXAAXYZFBIV3TBULYRFEWQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.601267617 (125h42m3.954052744s) [noise=true , address=104.36.82.6:4875 , piece_id=K3MQHDJ7BQ7QKBFY26U6XXNHMHMITUAW73SENK5XZCCGP5BMGCYQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.601338540 (125h42m3.954123668s) [noise=true , address=104.36.82.9:4007 , piece_id=AEGR6QK3SJTGSHHPVSJHTFJXNANSGNUA2ZIWEG3XHE5XSHQU2SQA, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.601701652 (125h42m3.954486791s) [noise=true , address=104.36.82.13:4187 , piece_id=4WYLRG25O5MREEZAI4RPTOTTTCDQLWUX7PRQGBRKOAZADW3AL5AA, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.601765142 (125h42m3.95455027s) [noise=true , address=104.36.82.7:4912 , piece_id=FS2EN4BM234GJL4YDODWRZVL77Z3BBSJFUQCFHLEHR224FMKLC2A, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.602663972 (125h42m3.955449101s) [noise=true , address=77.90.31.17:4770 , piece_id=RGYLLEBJSTESSSVA7UZD6V4FD32OS3I2Q7ITF23LWCSR34HWP5WQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.602815948 (125h42m3.955601086s) [noise=true , address=45.91.116.5:4297 , piece_id=VCD2L2LRDRDL3RL35XO3S5HAHLDESR2DY5F2V3SDY3PPDO3J63GA, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.603514982 (125h42m3.956300141s) [noise=true , address=45.91.116.8:4047 , piece_id=SMRGHSUUE5IE7PU4UWOOZUS5U6LFOGJNEWB4IUAJVESLGDYOSVTQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.603574185 (125h42m3.956359313s) [noise=true , address=45.91.116.13:4154 , piece_id=ISYL6B7DUA4ODXTRRVITR2BCXUVQGAC4LFDKBQSYF5AHX2HYHBCA, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.604871624 (125h42m3.957656762s) [noise=true , address=45.91.116.6:4974 , piece_id=HRFFQ22SADG7PTKGCNFIQ3VSKCBHG3JJDX6VLIR5MSR3T7ZHTP2A, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.605228004 (125h42m3.958013142s) [noise=true , address=104.36.82.10:4077 , piece_id=LZCEBG5HO3IJWP3GPIATPLXXXDMOQVXHMHKT2HVFWHGJU7E42FNQ, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.606132837 (125h42m3.958917964s) [noise=true , address=45.91.116.10:4083 , piece_id=5AMQUYP7LMRVG22OENG66EV53526GX4DULTG3OQTNR2STDTYID6A, seg_pos={0 0}, upload=110833]: Uploading piece... Sep 3 19:50:28.734930601 (125h42m4.087715719s) [upload=110833]: Next returned nil segment Sep 3 19:50:28.734943967 (125h42m4.087729075s) [upload=110833]: Waiting for error group managing segments... Sep 3 19:50:28.743859193 (125h42m4.096644311s) [noise=true , address=45.91.116.6:4974 , piece_id=HRFFQ22SADG7PTKGCNFIQ3VSKCBHG3JJDX6VLIR5MSR3T7ZHTP2A, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.744038891 (125h42m4.09682402s) [noise=true , address=45.91.116.5:4297 , piece_id=VCD2L2LRDRDL3RL35XO3S5HAHLDESR2DY5F2V3SDY3PPDO3J63GA, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.744641605 (125h42m4.097426713s) [noise=true , address=45.91.116.6:4975 , piece_id=QSNIWRITJ2UKMDYUJOJG3OJAJW2O4X76R6G5HCZ3BA4UZWVHBGEQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.744758946 (125h42m4.097544074s) [noise=true , address=77.90.31.17:4770 , piece_id=RGYLLEBJSTESSSVA7UZD6V4FD32OS3I2Q7ITF23LWCSR34HWP5WQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.745191980 (125h42m4.097977098s) [noise=true , address=45.91.116.12:4225 , piece_id=5OV6BI6KAO3KP5URFJYGRBQHY3XFNJNY52DMB5O3RBSL4XU4KS4Q, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.745786588 (125h42m4.098571716s) [noise=true , address=45.91.116.10:4083 , piece_id=5AMQUYP7LMRVG22OENG66EV53526GX4DULTG3OQTNR2STDTYID6A, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.746582645 (125h42m4.099367763s) [noise=true , address=45.91.116.4:4937 , piece_id=VFILXM44M7EU7XZWZTBEETATMMFA7IBXAAXYZFBIV3TBULYRFEWQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.756988735 (125h42m4.109773854s) [noise=true , address=45.91.116.8:4047 , piece_id=SMRGHSUUE5IE7PU4UWOOZUS5U6LFOGJNEWB4IUAJVESLGDYOSVTQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.759436769 (125h42m4.112221897s) [noise=true , address=45.91.116.9:4261 , piece_id=67JORI3KBKETLOSVQQWDVBBYPN7LSKAQQOXUMUWATHUGBIMCWA6Q, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.775962409 (125h42m4.128747547s) [noise=true , address=45.91.116.13:4154 , piece_id=ISYL6B7DUA4ODXTRRVITR2BCXUVQGAC4LFDKBQSYF5AHX2HYHBCA, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:28.896587655 (125h42m4.249372794s) [noise=true , address=77.90.31.16:4733 , piece_id=HOFKXNJBOJ27QNAWARHZSRA2F7R4XN2VAGY4TRLQ4BLQ6SZNBEMA, seg_pos={0 0}, upload=110833]: Done uploading piece. err: <nil> Sep 3 19:50:58.557541379 (125h42m33.910326487s) [noise=true , address=104.36.82.9:4007 , piece_id=AEGR6QK3SJTGSHHPVSJHTFJXNANSGNUA2ZIWEG3XHE5XSHQU2SQA, seg_pos={0 0}, upload=110833]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: context deadline exceeded; context deadline exceeded Sep 3 19:50:58.557639383 (125h42m33.910424491s) [noise=true , address=104.36.82.7:4912 , piece_id=FS2EN4BM234GJL4YDODWRZVL77Z3BBSJFUQCFHLEHR224FMKLC2A, seg_pos={0 0}, upload=110833]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 5.161.89.252:8574->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:50:58.557659522 (125h42m33.91044463s) [noise=true , address=104.36.82.13:4187 , piece_id=4WYLRG25O5MREEZAI4RPTOTTTCDQLWUX7PRQGBRKOAZADW3AL5AA, seg_pos={0 0}, upload=110833]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Sep 3 19:50:58.557734222 (125h42m33.91051933s) [noise=true , address=104.36.82.6:4875 , piece_id=K3MQHDJ7BQ7QKBFY26U6XXNHMHMITUAW73SENK5XZCCGP5BMGCYQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:50:58.557832475 (125h42m33.910617584s) [noise=true , address=104.36.82.10:4077 , piece_id=LZCEBG5HO3IJWP3GPIATPLXXXDMOQVXHMHKT2HVFWHGJU7E42FNQ, seg_pos={0 0}, upload=110833]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:50:58.557867893 (125h42m33.910653001s) [seg_pos={0 0}, upload=110833]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 3 19:50:58.572363874 (125h42m33.925148982s) [upload=110833]: Done uploading segments... Sep 3 19:50:58.572377410 (125h42m33.925162508s) [upload=110833]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.016973208s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X......................X.] duration: 655.02845ms FAIL (2025-09-03 19:50:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 655.079114ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X......................X] duration: 30.001238893s FAIL (2025-09-03 19:50:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001258499s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 374.005711ms FAIL (2025-09-03 19:49:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Sep 3 19:48:30.671750096 (125h40m6.024535204s) [upload=110805]: Starting upload Sep 3 19:48:30.671807253 (125h40m6.024592361s) [upload=110805]: Done starting upload Sep 3 19:48:30.671903123 (125h40m6.024688261s) [upload=110805]: Uploading segments... Sep 3 19:48:30.672089273 (125h40m6.024874371s) [upload=110805]: Got next segment. Inline: false Sep 3 19:48:30.672126083 (125h40m6.024911191s) [upload=110805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:48:30.672131103 (125h40m6.024916211s) [upload=110805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:48:30.712899134 (125h40m6.065684272s) [seg_pos={0 0}, upload=110805]: Begin upload segment... Sep 3 19:48:30.712947013 (125h40m6.065732142s) [seg_pos={0 0}, upload=110805]: Done begin upload segment. Sep 3 19:48:30.712981348 (125h40m6.065766476s) [noise=true , address=104.36.82.8:4971 , piece_id=GRBXCLGKWW5BYX2BY5BIQNSMSLW66NFBDO5XOJP32BINMF4SNTEQ, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.713049536 (125h40m6.065834644s) [noise=true , address=104.36.82.5:4839 , piece_id=WKGDCKTROGELIKVWX7ZAXHKIZTDEE7KSSNTBTNWF7DPOBMQ5CEOA, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.713434420 (125h40m6.066219548s) [noise=true , address=104.36.82.10:4077 , piece_id=KLO4ZMTMDLNH2EJCRH3QKSVVXIOVR4BQT2KOD2HMVSUQR5IZETMA, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.713595222 (125h40m6.066380351s) [noise=true , address=45.91.116.5:4299 , piece_id=UUZREM56TD3SFLSXIEYDDEHRVQDXBWRE7GZGT6OL4B2SDAS7IYGQ, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.713856024 (125h40m6.066641132s) [noise=true , address=45.91.116.8:4047 , piece_id=AU66JJ3XDZOUYOEHBHOGLM2QF6ETYXVY3IBKYXJKRFQB2JTGAJ3A, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.714149645 (125h40m6.066934783s) [noise=true , address=45.91.116.11:4117 , piece_id=ZBP4ABBFBRRK433EIJG4BBG4GMEWHI6EBNMJ2W6FN25RDG2YBZ7Q, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.714194890 (125h40m6.066979998s) [noise=true , address=77.90.31.13:4628 , piece_id=Z33RQLJSIBXPIX35IZW5H4INFVU6GLIITYKYYOBX4FHORWJPSQ4Q, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.714571628 (125h40m6.067356757s) [noise=true , address=104.36.82.4:4804 , piece_id=EZ7CDSEYOWYQKNI7LGRMIWWJDCN3GVOWXPQW6IECQDZS4GTNJGNA, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.714950821 (125h40m6.06773595s) [noise=true , address=45.91.116.6:4974 , piece_id=XAPNWFQCYQ25CNNLE3EN2HFHOP2NBDX5HJMVJE6C4DIJLFJDP7ZA, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.715448398 (125h40m6.068233516s) [noise=true , address=104.36.82.2:4730 , piece_id=BEITPNRKSBS6LZFU7CQHA4HA4D6CM725YVOEFRDP3IVPL4OQRWKA, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.715833232 (125h40m6.06861836s) [noise=true , address=104.36.82.6:4873 , piece_id=FZIRQ6GIH4EMG6R5BHC45N6ZQICACEI5TISRRHVMOTVRKJJVMGUA, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.716217815 (125h40m6.069002943s) [noise=true , address=104.36.82.9:4007 , piece_id=6PLUNTXYX3X73BSUY2JVSMCQFKA6AHJD42VK4HXWVK4I6Y4EJSYA, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.716608199 (125h40m6.069393327s) [noise=true , address=45.91.116.4:4937 , piece_id=ISJXRS3CMOAYHJRRK5CWC3EULUYMDXP5T2TZCYK467T62EYIWDIQ, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.716995959 (125h40m6.069781077s) [noise=true , address=77.90.31.18:4805 , piece_id=54VDP6FKNVZ266DADNKKFMDH3WPCPWXJQVNRNSNHYG6GLWM25H2Q, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.717436256 (125h40m6.070221384s) [noise=true , address=104.36.82.13:4186 , piece_id=B4CTVB3WEJTEIKEXDEH6UK3I7MDAH4YZMDUDRQIAORWVYXBODLSA, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.717963839 (125h40m6.070748967s) [noise=true , address=45.91.116.12:4225 , piece_id=7Z2SJD45BBF5W5JVE4GJNS24UUYY6NTIX33TBN6QJFLU62MAXZGQ, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.809552667 (125h40m6.162337795s) [upload=110805]: Next returned nil segment Sep 3 19:48:30.809568296 (125h40m6.162353424s) [upload=110805]: Waiting for error group managing segments... Sep 3 19:48:30.821225429 (125h40m6.174010527s) [noise=true , address=45.91.116.12:4225 , piece_id=7Z2SJD45BBF5W5JVE4GJNS24UUYY6NTIX33TBN6QJFLU62MAXZGQ, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:48:30.823665858 (125h40m6.176450986s) [noise=true , address=45.91.116.6:4974 , piece_id=XAPNWFQCYQ25CNNLE3EN2HFHOP2NBDX5HJMVJE6C4DIJLFJDP7ZA, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:48:30.827885434 (125h40m6.180670542s) [noise=true , address=104.36.82.10:4077 , piece_id=KLO4ZMTMDLNH2EJCRH3QKSVVXIOVR4BQT2KOD2HMVSUQR5IZETMA, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:48:30.832691362 (125h40m6.18547647s) [noise=true , address=45.91.116.8:4047 , piece_id=AU66JJ3XDZOUYOEHBHOGLM2QF6ETYXVY3IBKYXJKRFQB2JTGAJ3A, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:48:30.834611533 (125h40m6.187396641s) [noise=true , address=45.91.116.5:4299 , piece_id=UUZREM56TD3SFLSXIEYDDEHRVQDXBWRE7GZGT6OL4B2SDAS7IYGQ, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:48:30.850594532 (125h40m6.20337964s) [noise=true , address=45.91.116.4:4937 , piece_id=ISJXRS3CMOAYHJRRK5CWC3EULUYMDXP5T2TZCYK467T62EYIWDIQ, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:48:30.892785670 (125h40m6.245570798s) [noise=true , address=45.91.116.11:4117 , piece_id=ZBP4ABBFBRRK433EIJG4BBG4GMEWHI6EBNMJ2W6FN25RDG2YBZ7Q, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:48:30.900132788 (125h40m6.252917915s) [noise=true , address=77.90.31.13:4628 , piece_id=Z33RQLJSIBXPIX35IZW5H4INFVU6GLIITYKYYOBX4FHORWJPSQ4Q, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:48:31.405933890 (125h40m6.758719018s) [noise=true , address=77.90.31.18:4805 , piece_id=54VDP6FKNVZ266DADNKKFMDH3WPCPWXJQVNRNSNHYG6GLWM25H2Q, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:49:00.657389261 (125h40m36.010174389s) [noise=true , address=104.36.82.4:4804 , piece_id=EZ7CDSEYOWYQKNI7LGRMIWWJDCN3GVOWXPQW6IECQDZS4GTNJGNA, seg_pos={0 0}, upload=110805]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:49:00.657511811 (125h40m36.01029692s) [noise=true , address=104.36.82.2:4730 , piece_id=BEITPNRKSBS6LZFU7CQHA4HA4D6CM725YVOEFRDP3IVPL4OQRWKA, seg_pos={0 0}, upload=110805]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:49:00.657584639 (125h40m36.010369757s) [noise=true , address=104.36.82.6:4873 , piece_id=FZIRQ6GIH4EMG6R5BHC45N6ZQICACEI5TISRRHVMOTVRKJJVMGUA, seg_pos={0 0}, upload=110805]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:49:00.657674177 (125h40m36.010459305s) [noise=true , address=104.36.82.13:4186 , piece_id=B4CTVB3WEJTEIKEXDEH6UK3I7MDAH4YZMDUDRQIAORWVYXBODLSA, seg_pos={0 0}, upload=110805]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:49:00.657774957 (125h40m36.010560075s) [noise=true , address=104.36.82.5:4839 , piece_id=WKGDCKTROGELIKVWX7ZAXHKIZTDEE7KSSNTBTNWF7DPOBMQ5CEOA, seg_pos={0 0}, upload=110805]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 5.161.89.252:49676->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:49:00.657868072 (125h40m36.01065319s) [noise=true , address=104.36.82.8:4971 , piece_id=GRBXCLGKWW5BYX2BY5BIQNSMSLW66NFBDO5XOJP32BINMF4SNTEQ, seg_pos={0 0}, upload=110805]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:49:00.657954283 (125h40m36.010739402s) [noise=true , address=104.36.82.9:4007 , piece_id=6PLUNTXYX3X73BSUY2JVSMCQFKA6AHJD42VK4HXWVK4I6Y4EJSYA, seg_pos={0 0}, upload=110805]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 5.161.89.252:29090->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 19:49:00.657975724 (125h40m36.010760832s) [seg_pos={0 0}, upload=110805]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 3 19:49:00.680373256 (125h40m36.033158364s) [upload=110805]: Done uploading segments... Sep 3 19:49:00.680394546 (125h40m36.033179664s) [upload=110805]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 374.029464ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 496.053009ms FAIL (2025-09-03 19:49:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 496.073336ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.025031129s FAIL (2025-09-03 19:49:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Sep 3 19:48:30.671750096 (125h40m6.024535204s) [upload=110805]: Starting upload Sep 3 19:48:30.671807253 (125h40m6.024592361s) [upload=110805]: Done starting upload Sep 3 19:48:30.671903123 (125h40m6.024688261s) [upload=110805]: Uploading segments... Sep 3 19:48:30.672089273 (125h40m6.024874371s) [upload=110805]: Got next segment. Inline: false Sep 3 19:48:30.672126083 (125h40m6.024911191s) [upload=110805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:48:30.672131103 (125h40m6.024916211s) [upload=110805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:48:30.712899134 (125h40m6.065684272s) [seg_pos={0 0}, upload=110805]: Begin upload segment... Sep 3 19:48:30.712947013 (125h40m6.065732142s) [seg_pos={0 0}, upload=110805]: Done begin upload segment. Sep 3 19:48:30.712981348 (125h40m6.065766476s) [noise=true , address=104.36.82.8:4971 , piece_id=GRBXCLGKWW5BYX2BY5BIQNSMSLW66NFBDO5XOJP32BINMF4SNTEQ, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.713049536 (125h40m6.065834644s) [noise=true , address=104.36.82.5:4839 , piece_id=WKGDCKTROGELIKVWX7ZAXHKIZTDEE7KSSNTBTNWF7DPOBMQ5CEOA, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.713434420 (125h40m6.066219548s) [noise=true , address=104.36.82.10:4077 , piece_id=KLO4ZMTMDLNH2EJCRH3QKSVVXIOVR4BQT2KOD2HMVSUQR5IZETMA, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.713595222 (125h40m6.066380351s) [noise=true , address=45.91.116.5:4299 , piece_id=UUZREM56TD3SFLSXIEYDDEHRVQDXBWRE7GZGT6OL4B2SDAS7IYGQ, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.713856024 (125h40m6.066641132s) [noise=true , address=45.91.116.8:4047 , piece_id=AU66JJ3XDZOUYOEHBHOGLM2QF6ETYXVY3IBKYXJKRFQB2JTGAJ3A, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.714149645 (125h40m6.066934783s) [noise=true , address=45.91.116.11:4117 , piece_id=ZBP4ABBFBRRK433EIJG4BBG4GMEWHI6EBNMJ2W6FN25RDG2YBZ7Q, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.714194890 (125h40m6.066979998s) [noise=true , address=77.90.31.13:4628 , piece_id=Z33RQLJSIBXPIX35IZW5H4INFVU6GLIITYKYYOBX4FHORWJPSQ4Q, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.714571628 (125h40m6.067356757s) [noise=true , address=104.36.82.4:4804 , piece_id=EZ7CDSEYOWYQKNI7LGRMIWWJDCN3GVOWXPQW6IECQDZS4GTNJGNA, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.714950821 (125h40m6.06773595s) [noise=true , address=45.91.116.6:4974 , piece_id=XAPNWFQCYQ25CNNLE3EN2HFHOP2NBDX5HJMVJE6C4DIJLFJDP7ZA, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.715448398 (125h40m6.068233516s) [noise=true , address=104.36.82.2:4730 , piece_id=BEITPNRKSBS6LZFU7CQHA4HA4D6CM725YVOEFRDP3IVPL4OQRWKA, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.715833232 (125h40m6.06861836s) [noise=true , address=104.36.82.6:4873 , piece_id=FZIRQ6GIH4EMG6R5BHC45N6ZQICACEI5TISRRHVMOTVRKJJVMGUA, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.716217815 (125h40m6.069002943s) [noise=true , address=104.36.82.9:4007 , piece_id=6PLUNTXYX3X73BSUY2JVSMCQFKA6AHJD42VK4HXWVK4I6Y4EJSYA, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.716608199 (125h40m6.069393327s) [noise=true , address=45.91.116.4:4937 , piece_id=ISJXRS3CMOAYHJRRK5CWC3EULUYMDXP5T2TZCYK467T62EYIWDIQ, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.716995959 (125h40m6.069781077s) [noise=true , address=77.90.31.18:4805 , piece_id=54VDP6FKNVZ266DADNKKFMDH3WPCPWXJQVNRNSNHYG6GLWM25H2Q, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.717436256 (125h40m6.070221384s) [noise=true , address=104.36.82.13:4186 , piece_id=B4CTVB3WEJTEIKEXDEH6UK3I7MDAH4YZMDUDRQIAORWVYXBODLSA, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.717963839 (125h40m6.070748967s) [noise=true , address=45.91.116.12:4225 , piece_id=7Z2SJD45BBF5W5JVE4GJNS24UUYY6NTIX33TBN6QJFLU62MAXZGQ, seg_pos={0 0}, upload=110805]: Uploading piece... Sep 3 19:48:30.809552667 (125h40m6.162337795s) [upload=110805]: Next returned nil segment Sep 3 19:48:30.809568296 (125h40m6.162353424s) [upload=110805]: Waiting for error group managing segments... Sep 3 19:48:30.821225429 (125h40m6.174010527s) [noise=true , address=45.91.116.12:4225 , piece_id=7Z2SJD45BBF5W5JVE4GJNS24UUYY6NTIX33TBN6QJFLU62MAXZGQ, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:48:30.823665858 (125h40m6.176450986s) [noise=true , address=45.91.116.6:4974 , piece_id=XAPNWFQCYQ25CNNLE3EN2HFHOP2NBDX5HJMVJE6C4DIJLFJDP7ZA, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:48:30.827885434 (125h40m6.180670542s) [noise=true , address=104.36.82.10:4077 , piece_id=KLO4ZMTMDLNH2EJCRH3QKSVVXIOVR4BQT2KOD2HMVSUQR5IZETMA, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:48:30.832691362 (125h40m6.18547647s) [noise=true , address=45.91.116.8:4047 , piece_id=AU66JJ3XDZOUYOEHBHOGLM2QF6ETYXVY3IBKYXJKRFQB2JTGAJ3A, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:48:30.834611533 (125h40m6.187396641s) [noise=true , address=45.91.116.5:4299 , piece_id=UUZREM56TD3SFLSXIEYDDEHRVQDXBWRE7GZGT6OL4B2SDAS7IYGQ, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:48:30.850594532 (125h40m6.20337964s) [noise=true , address=45.91.116.4:4937 , piece_id=ISJXRS3CMOAYHJRRK5CWC3EULUYMDXP5T2TZCYK467T62EYIWDIQ, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:48:30.892785670 (125h40m6.245570798s) [noise=true , address=45.91.116.11:4117 , piece_id=ZBP4ABBFBRRK433EIJG4BBG4GMEWHI6EBNMJ2W6FN25RDG2YBZ7Q, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:48:30.900132788 (125h40m6.252917915s) [noise=true , address=77.90.31.13:4628 , piece_id=Z33RQLJSIBXPIX35IZW5H4INFVU6GLIITYKYYOBX4FHORWJPSQ4Q, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:48:31.405933890 (125h40m6.758719018s) [noise=true , address=77.90.31.18:4805 , piece_id=54VDP6FKNVZ266DADNKKFMDH3WPCPWXJQVNRNSNHYG6GLWM25H2Q, seg_pos={0 0}, upload=110805]: Done uploading piece. err: <nil> Sep 3 19:49:00.657389261 (125h40m36.010174389s) [noise=true , address=104.36.82.4:4804 , piece_id=EZ7CDSEYOWYQKNI7LGRMIWWJDCN3GVOWXPQW6IECQDZS4GTNJGNA, seg_pos={0 0}, upload=110805]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:49:00.657511811 (125h40m36.01029692s) [noise=true , address=104.36.82.2:4730 , piece_id=BEITPNRKSBS6LZFU7CQHA4HA4D6CM725YVOEFRDP3IVPL4OQRWKA, seg_pos={0 0}, upload=110805]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:49:00.657584639 (125h40m36.010369757s) [noise=true , address=104.36.82.6:4873 , piece_id=FZIRQ6GIH4EMG6R5BHC45N6ZQICACEI5TISRRHVMOTVRKJJVMGUA, seg_pos={0 0}, upload=110805]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:49:00.657674177 (125h40m36.010459305s) [noise=true , address=104.36.82.13:4186 , piece_id=B4CTVB3WEJTEIKEXDEH6UK3I7MDAH4YZMDUDRQIAORWVYXBODLSA, seg_pos={0 0}, upload=110805]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:49:00.657774957 (125h40m36.010560075s) [noise=true , address=104.36.82.5:4839 , piece_id=WKGDCKTROGELIKVWX7ZAXHKIZTDEE7KSSNTBTNWF7DPOBMQ5CEOA, seg_pos={0 0}, upload=110805]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 5.161.89.252:49676->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:49:00.657868072 (125h40m36.01065319s) [noise=true , address=104.36.82.8:4971 , piece_id=GRBXCLGKWW5BYX2BY5BIQNSMSLW66NFBDO5XOJP32BINMF4SNTEQ, seg_pos={0 0}, upload=110805]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:49:00.657954283 (125h40m36.010739402s) [noise=true , address=104.36.82.9:4007 , piece_id=6PLUNTXYX3X73BSUY2JVSMCQFKA6AHJD42VK4HXWVK4I6Y4EJSYA, seg_pos={0 0}, upload=110805]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 5.161.89.252:29090->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 19:49:00.657975724 (125h40m36.010760832s) [seg_pos={0 0}, upload=110805]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 3 19:49:00.680373256 (125h40m36.033158364s) [upload=110805]: Done uploading segments... Sep 3 19:49:00.680394546 (125h40m36.033179664s) [upload=110805]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.0250527s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.01775909s FAIL (2025-09-03 19:48:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 19:47:56.281381414 (125h39m31.634166591s) [upload=110799]: Starting upload Sep 3 19:47:56.281542346 (125h39m31.634327474s) [upload=110799]: Done starting upload Sep 3 19:47:56.281672562 (125h39m31.63445767s) [upload=110799]: Uploading segments... Sep 3 19:47:56.281944653 (125h39m31.634729781s) [upload=110799]: Got next segment. Inline: false Sep 3 19:47:56.281987914 (125h39m31.634773022s) [upload=110799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:47:56.281994226 (125h39m31.634779334s) [upload=110799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:47:56.306434831 (125h39m31.659219939s) [seg_pos={0 0}, upload=110799]: Begin upload segment... Sep 3 19:47:56.306488642 (125h39m31.65927375s) [seg_pos={0 0}, upload=110799]: Done begin upload segment. Sep 3 19:47:56.306514050 (125h39m31.659299158s) [noise=true , address=45.91.116.7:4011 , piece_id=J6PXMMWZASELTYU7YNIQPMNUFIGGHCQDMF32JYD5XOAIN4NBPWHQ, seg_pos={0 0}, upload=110799]: Uploading piece... Sep 3 19:47:56.306526002 (125h39m31.659311131s) [noise=true , address=45.91.116.7:4009 , piece_id=6MABGVSSD6P2MOLW3MY3XLBKESE5SX7E6QMFVEOD2KA7R2BFUE5Q, seg_pos={0 0}, upload=110799]: Uploading piece... Sep 3 19:47:56.306814585 (125h39m31.659599693s) [noise=true , address=104.36.82.12:4151 , piece_id=UFVAIBRNLNMVRXXCTAZIA56AP7MEBDJVGGPRGSXJMVZX4EEGKMEQ, seg_pos={0 0}, upload=110799]: Uploading piece... Sep 3 19:47:56.307034849 (125h39m31.659819957s) [noise=true , address=77.90.31.19:4842 , piece_id=KE3Y6VVZHVNNMVQT3SAHRXUBLA2V6BZYUWRCIE3Y2OOU3ZAH2QZQ, seg_pos={0 0}, upload=110799]: Uploading piece... Sep 3 19:47:56.307173961 (125h39m31.659959088s) [noise=true , address=45.91.116.9:4263 , piece_id=NAFC2HIRF2N5HLXRIGFVM5F463YL6OYBGTP3XO5YOCRMSCKMJDVQ, seg_pos={0 0}, upload=110799]: Uploading piece... Sep 3 19:47:56.307258259 (125h39m31.660043367s) [noise=true , address=77.90.31.13:4628 , piece_id=JFPKZCKMNPW2GGAATQJSHUJR3C756N4R3DMZOPV2EE7TGJENO76A, seg_pos={0 0}, upload=110799]: Uploading piece... Sep 3 19:47:56.307506877 (125h39m31.660291975s) [noise=true , address=45.91.116.4:4938 , piece_id=SOHQPLMX5TSFYRECFDC5D2V5AFK5EYRWK7KMAVQD3X3T75ZOY2IQ, seg_pos={0 0}, upload=110799]: Uploading piece... Sep 3 19:47:56.307769139 (125h39m31.660554248s) [noise=true , address=45.91.116.12:4227 , piece_id=UYMJ3IJCZ2PGXEFP6XL7LUOVUB6OZ3PXSLSSMUSTFXUM7F6ASU6A, seg_pos={0 0}, upload=110799]: Uploading piece... Sep 3 19:47:56.308044338 (125h39m31.660829446s) [noise=true , address=104.36.82.7:4910 , piece_id=QJ2NMCV2HQXYVNQP2SMVPN4VQEWCYBRJR4C4GIVJGWTERI6XOHMQ, seg_pos={0 0}, upload=110799]: Uploading piece... Sep 3 19:47:56.308320767 (125h39m31.661105875s) [noise=true , address=45.91.116.13:4155 , piece_id=XWLJDQ6JYPMYLW2BC7WHYF7EVIZOGITTVOQZQSUH3AAXEU5E4ALQ, seg_pos={0 0}, upload=110799]: Uploading piece... Sep 3 19:47:56.308601906 (125h39m31.661387014s) [noise=true , address=77.90.31.14:4662 , piece_id=K6H4H64XJO6VX4TOISJ2M6GB2FLDFLZ3A4IJJZQJAQLFC2Z35RMQ, seg_pos={0 0}, upload=110799]: Uploading piece... Sep 3 19:47:56.309180976 (125h39m31.661966084s) [noise=true , address=104.36.82.8:4972 , piece_id=MWMBHEPUSH4EMQRC6RBI5ZNEUDTQXSTBQQQBZGB7VV4OEPOP3YSQ, seg_pos={0 0}, upload=110799]: Uploading piece... Sep 3 19:47:56.309728063 (125h39m31.662513172s) [noise=true , address=77.90.31.15:4699 , piece_id=XAVS4UFMZI7SJ5S4RMNFVFNSAK3YG5AOJP3QGGXWJGK5WRNXWM5Q, seg_pos={0 0}, upload=110799]: Uploading piece... Sep 3 19:47:56.310544470 (125h39m31.663329568s) [noise=true , address=77.90.31.21:4878 , piece_id=QZ5BGADMDFOMSA3JF5OXXBF4NWMANXQ2ZO56U25YNAJZYBB2YDCA, seg_pos={0 0}, upload=110799]: Uploading piece... Sep 3 19:47:56.311445684 (125h39m31.664230822s) [noise=true , address=104.36.82.9:4005 , piece_id=P36QFESXQON5MXUZ2A6E6MP3EN4Q7JMFYXSQSUPQBGVH65JWTXGQ, seg_pos={0 0}, upload=110799]: Uploading piece... Sep 3 19:47:56.312327553 (125h39m31.665112661s) [noise=true , address=104.36.82.6:4875 , piece_id=ITBJZNA3AS3YIP5HN2LOQTQNC7ED2DOOAEVN63DBKZSLDGO7W2DA, seg_pos={0 0}, upload=110799]: Uploading piece... Sep 3 19:47:56.429909204 (125h39m31.782694302s) [upload=110799]: Next returned nil segment Sep 3 19:47:56.429922610 (125h39m31.782707718s) [upload=110799]: Waiting for error group managing segments... Sep 3 19:47:56.439174459 (125h39m31.791959558s) [noise=true , address=45.91.116.7:4009 , piece_id=6MABGVSSD6P2MOLW3MY3XLBKESE5SX7E6QMFVEOD2KA7R2BFUE5Q, seg_pos={0 0}, upload=110799]: Done uploading piece. err: <nil> Sep 3 19:47:56.439331685 (125h39m31.792116793s) [noise=true , address=45.91.116.9:4263 , piece_id=NAFC2HIRF2N5HLXRIGFVM5F463YL6OYBGTP3XO5YOCRMSCKMJDVQ, seg_pos={0 0}, upload=110799]: Done uploading piece. err: <nil> Sep 3 19:47:56.445239124 (125h39m31.798024223s) [noise=true , address=45.91.116.13:4155 , piece_id=XWLJDQ6JYPMYLW2BC7WHYF7EVIZOGITTVOQZQSUH3AAXEU5E4ALQ, seg_pos={0 0}, upload=110799]: Done uploading piece. err: <nil> Sep 3 19:47:56.446720621 (125h39m31.799505729s) [noise=true , address=104.36.82.12:4151 , piece_id=UFVAIBRNLNMVRXXCTAZIA56AP7MEBDJVGGPRGSXJMVZX4EEGKMEQ, seg_pos={0 0}, upload=110799]: Done uploading piece. err: <nil> Sep 3 19:47:56.448365885 (125h39m31.801150983s) [noise=true , address=45.91.116.12:4227 , piece_id=UYMJ3IJCZ2PGXEFP6XL7LUOVUB6OZ3PXSLSSMUSTFXUM7F6ASU6A, seg_pos={0 0}, upload=110799]: Done uploading piece. err: <nil> Sep 3 19:47:56.465984220 (125h39m31.818769318s) [noise=true , address=77.90.31.21:4878 , piece_id=QZ5BGADMDFOMSA3JF5OXXBF4NWMANXQ2ZO56U25YNAJZYBB2YDCA, seg_pos={0 0}, upload=110799]: Done uploading piece. err: <nil> Sep 3 19:47:56.471226419 (125h39m31.824011557s) [noise=true , address=77.90.31.14:4662 , piece_id=K6H4H64XJO6VX4TOISJ2M6GB2FLDFLZ3A4IJJZQJAQLFC2Z35RMQ, seg_pos={0 0}, upload=110799]: Done uploading piece. err: <nil> Sep 3 19:47:56.474260985 (125h39m31.827046083s) [noise=true , address=45.91.116.4:4938 , piece_id=SOHQPLMX5TSFYRECFDC5D2V5AFK5EYRWK7KMAVQD3X3T75ZOY2IQ, seg_pos={0 0}, upload=110799]: Done uploading piece. err: <nil> Sep 3 19:47:56.474679682 (125h39m31.827464821s) [noise=true , address=45.91.116.7:4011 , piece_id=J6PXMMWZASELTYU7YNIQPMNUFIGGHCQDMF32JYD5XOAIN4NBPWHQ, seg_pos={0 0}, upload=110799]: Done uploading piece. err: <nil> Sep 3 19:47:56.484008556 (125h39m31.836793655s) [noise=true , address=77.90.31.15:4699 , piece_id=XAVS4UFMZI7SJ5S4RMNFVFNSAK3YG5AOJP3QGGXWJGK5WRNXWM5Q, seg_pos={0 0}, upload=110799]: Done uploading piece. err: <nil> Sep 3 19:47:56.484531771 (125h39m31.837316899s) [noise=true , address=77.90.31.19:4842 , piece_id=KE3Y6VVZHVNNMVQT3SAHRXUBLA2V6BZYUWRCIE3Y2OOU3ZAH2QZQ, seg_pos={0 0}, upload=110799]: Done uploading piece. err: <nil> Sep 3 19:47:56.508893628 (125h39m31.861678736s) [noise=true , address=77.90.31.13:4628 , piece_id=JFPKZCKMNPW2GGAATQJSHUJR3C756N4R3DMZOPV2EE7TGJENO76A, seg_pos={0 0}, upload=110799]: Done uploading piece. err: <nil> Sep 3 19:48:26.268624915 (125h40m1.621410023s) [noise=true , address=104.36.82.6:4875 , piece_id=ITBJZNA3AS3YIP5HN2LOQTQNC7ED2DOOAEVN63DBKZSLDGO7W2DA, seg_pos={0 0}, upload=110799]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 5.161.89.252:24482->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:48:26.268671803 (125h40m1.621456911s) [noise=true , address=104.36.82.8:4972 , piece_id=MWMBHEPUSH4EMQRC6RBI5ZNEUDTQXSTBQQQBZGB7VV4OEPOP3YSQ, seg_pos={0 0}, upload=110799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 5.161.89.252:18422->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:48:26.268681240 (125h40m1.621466348s) [noise=true , address=104.36.82.7:4910 , piece_id=QJ2NMCV2HQXYVNQP2SMVPN4VQEWCYBRJR4C4GIVJGWTERI6XOHMQ, seg_pos={0 0}, upload=110799]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:48:26.268736544 (125h40m1.621521652s) [noise=true , address=104.36.82.9:4005 , piece_id=P36QFESXQON5MXUZ2A6E6MP3EN4Q7JMFYXSQSUPQBGVH65JWTXGQ, seg_pos={0 0}, upload=110799]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 5.161.89.252:41626->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:48:26.268763424 (125h40m1.621548522s) [seg_pos={0 0}, upload=110799]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 19:48:26.284948574 (125h40m1.637733692s) [upload=110799]: Done uploading segments... Sep 3 19:48:26.284965676 (125h40m1.637750804s) [upload=110799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.017781923s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 407.960318ms FAIL (2025-09-03 19:48:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 407.984062ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 444.979759ms FAIL (2025-09-03 19:47:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 444.995398ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 466.671873ms FAIL (2025-09-03 19:47:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 466.690578ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 671.22476ms FAIL (2025-09-03 19:46:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 671.244235ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 398.229176ms FAIL (2025-09-03 19:45:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 398.251138ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 375.376807ms FAIL (2025-09-03 19:44:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 375.397487ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 636.721218ms FAIL (2025-09-03 19:43:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 636.743479ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 936.788474ms FAIL (2025-09-03 19:43:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 936.810886ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 579.16236ms FAIL (2025-09-03 19:42:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 579.180013ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 415.397243ms FAIL (2025-09-03 19:41:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 415.42272ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 465.823037ms FAIL (2025-09-03 19:40:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 465.845278ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 400.781427ms FAIL (2025-09-03 19:39:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 400.799552ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 396.240926ms FAIL (2025-09-03 19:38:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 396.474647ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 435.563168ms FAIL (2025-09-03 19:37:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 435.583648ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 443.993832ms FAIL (2025-09-03 19:37:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 444.016434ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 442.567742ms FAIL (2025-09-03 19:37:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 442.585856ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)