Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................XX.X..X.X..X...............................] duration: 369.583885ms FAIL (2025-09-03 20:21:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 369.609332ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...................X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX] duration: 30.000501613s FAIL (2025-09-03 20:21:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000513434s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................XX.X..X.X..X..............................] duration: 1.502105086s FAIL (2025-09-03 20:20:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.502120755s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....................X..X.XXX...XX.XX.XX..X...X.X.X.X.......X] duration: 30.001461772s FAIL (2025-09-03 20:20:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001478424s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....................X..X.XXX...XX.XX.XX..X...X.X.X.X.......] duration: 507.441108ms FAIL (2025-09-03 20:19:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 507.463039ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................XX.X..X.X..X.............................] duration: 465.523883ms FAIL (2025-09-03 20:19:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 465.550553ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......................X..X.XXX...XX.XX.XX..X...X.X.X.X......] duration: 489.065255ms FAIL (2025-09-03 20:18:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 489.087998ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................XX.X..X.X..X............................] duration: 474.037165ms FAIL (2025-09-03 20:18:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 474.062001ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X......................X..X.XXX...XX.XX.XX..X...X.X.X.X.....] duration: 452.775246ms FAIL (2025-09-03 20:17:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 452.793029ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................XX.X..X.X..X...........................] duration: 361.68565ms FAIL (2025-09-03 20:17:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 361.7129ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X......................X..X.XXX...XX.XX.XX..X...X.X.X.X....] duration: 491.094262ms FAIL (2025-09-03 20:17:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 491.120733ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................XX.X..X.X..X..........................] duration: 415.296152ms FAIL (2025-09-03 20:16:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 415.324606ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X......................X..X.XXX...XX.XX.XX..X...X.X.X.X...] duration: 514.236116ms FAIL (2025-09-03 20:16:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 514.268347ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................XX.X..X.X..X.........................] duration: 517.589241ms FAIL (2025-09-03 20:16:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 517.61496ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X......................X..X.XXX...XX.XX.XX..X...X.X.X.X..] duration: 484.529938ms FAIL (2025-09-03 20:15:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 484.553764ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................XX.X..X.X..X........................] duration: 633.293001ms FAIL (2025-09-03 20:15:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 633.323418ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................XX.X..X.X..X.......................] duration: 627.794571ms FAIL (2025-09-03 20:15:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 627.829105ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....X......................X..X.XXX...XX.XX.XX..X...X.X.X.X.] duration: 762.893045ms FAIL (2025-09-03 20:14:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 762.906731ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....X......................X..X.XXX...XX.XX.XX..X...X.X.X.X] duration: 30.001163206s FAIL (2025-09-03 20:14:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001178304s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................XX.X..X.X..X......................] duration: 363.737799ms FAIL (2025-09-03 20:14:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 363.760682ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......X......................X..X.XXX...XX.XX.XX..X...X.X.X.] duration: 430.756871ms FAIL (2025-09-03 20:13:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 430.774423ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................XX.X..X.X..X.....................] duration: 696.122462ms FAIL (2025-09-03 20:13:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 696.141498ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.......X......................X..X.XXX...XX.XX.XX..X...X.X.X] duration: 30.001152818s FAIL (2025-09-03 20:13:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001165342s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................XX.X..X.X..X....................] duration: 458.9708ms FAIL (2025-09-03 20:12:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 459.000657ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [........X......................X..X.XXX...XX.XX.XX..X...X.X.] duration: 452.286029ms FAIL (2025-09-03 20:12:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 452.304424ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.........X......................X..X.XXX...XX.XX.XX..X...X.X] duration: 30.000845461s FAIL (2025-09-03 20:12:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.0008617s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................XX.X..X.X..X...................] duration: 649.180662ms FAIL (2025-09-03 20:12:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 649.20083ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................XX.X..X.X..X..................] duration: 416.851578ms FAIL (2025-09-03 20:11:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 416.874782ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..........X......................X..X.XXX...XX.XX.XX..X...X.] duration: 528.381538ms FAIL (2025-09-03 20:11:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 528.708815ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................XX.X..X.X..X.................] duration: 558.308049ms FAIL (2025-09-03 20:11:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 558.333939ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...........X......................X..X.XXX...XX.XX.XX..X...X] duration: 30.000646551s FAIL (2025-09-03 20:10:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000658955s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................XX.X..X.X..X................] duration: 349.469916ms FAIL (2025-09-03 20:09:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 349.494454ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................XX.X..X.X..X...............] duration: 405.040495ms FAIL (2025-09-03 20:09:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 405.06968ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [............X......................X..X.XXX...XX.XX.XX..X...] duration: 408.312237ms FAIL (2025-09-03 20:09:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 408.332135ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................XX.X..X.X..X..............] duration: 352.206732ms FAIL (2025-09-03 20:08:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 352.230177ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.............X......................X..X.XXX...XX.XX.XX..X..] duration: 444.663924ms FAIL (2025-09-03 20:08:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 444.67751ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................XX.X..X.X..X.............] duration: 433.495931ms FAIL (2025-09-03 20:07:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 433.526327ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..............X......................X..X.XXX...XX.XX.XX..X.] duration: 413.424102ms FAIL (2025-09-03 20:07:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 413.446294ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............X......................X..X.XXX...XX.XX.XX..X] duration: 30.001224916s FAIL (2025-09-03 20:06:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001240936s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................................XX.X..X.X..X............] duration: 372.61762ms FAIL (2025-09-03 20:06:18 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 372.642346ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [................X......................X..X.XXX...XX.XX.XX..] duration: 2.076110408s FAIL (2025-09-03 20:06:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.076130455s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................XX.X..X.X..X...........] duration: 553.507934ms FAIL (2025-09-03 20:05:14 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 553.532139ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.................X......................X..X.XXX...XX.XX.XX.] duration: 907.613454ms FAIL (2025-09-03 20:05:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 907.636798ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..................X......................X..X.XXX...XX.XX.XX] duration: 30.002146379s FAIL (2025-09-03 20:04:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002162359s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................XX.X..X.X..X..........] duration: 432.364733ms FAIL (2025-09-03 20:04:22 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 432.503171ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...................X......................X..X.XXX...XX.XX.X] duration: 30.00045677s FAIL (2025-09-03 20:03:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00047351s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................................XX.X..X.X..X.........] duration: 399.301864ms FAIL (2025-09-03 20:03:36 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 399.316031ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................................XX.X..X.X..X........] duration: 362.549043ms FAIL (2025-09-03 20:03:03 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 362.570794ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....................X......................X..X.XXX...XX.XX.] duration: 426.925073ms FAIL (2025-09-03 20:02:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 426.949268ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................................XX.X..X.X..X.......] duration: 397.857578ms FAIL (2025-09-03 20:02:16 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 397.874359ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)