Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................XXX..........................] duration: 378.254122ms FAIL (2025-09-23 19:08:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 378.271234ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................XXX.........................] duration: 350.274184ms FAIL (2025-09-23 19:07:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 350.292027ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.X.......................] duration: 402.373154ms FAIL (2025-09-23 19:07:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 402.398512ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................XXX........................] duration: 420.260087ms FAIL (2025-09-23 19:07:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 420.282539ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.X......................] duration: 411.920958ms FAIL (2025-09-23 19:07:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 411.947317ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................XXX.......................] duration: 376.089377ms FAIL (2025-09-23 19:06:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 376.106759ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.X.....................] duration: 477.786038ms FAIL (2025-09-23 19:06:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 477.803361ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................XXX......................] duration: 414.500789ms FAIL (2025-09-23 19:06:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 414.536055ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................XXX.....................] duration: 458.736877ms FAIL (2025-09-23 19:05:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 458.758397ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.X....................] duration: 519.789998ms FAIL (2025-09-23 19:05:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 519.816849ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................XXX....................] duration: 407.804752ms FAIL (2025-09-23 19:05:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 408.14543ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................XXX...................] duration: 418.73691ms FAIL (2025-09-23 19:04:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 418.756557ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.X...................] duration: 440.682456ms FAIL (2025-09-23 19:04:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 440.704839ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................XXX..................] duration: 606.015562ms FAIL (2025-09-23 19:03:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 606.036492ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.X..................] duration: 517.424977ms FAIL (2025-09-23 19:03:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 517.449683ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................XXX.................] duration: 417.652348ms FAIL (2025-09-23 19:02:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 417.67496ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.X.................] duration: 733.213331ms FAIL (2025-09-23 19:01:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 733.255662ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.X................] duration: 333.512106ms FAIL (2025-09-23 19:01:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 333.539157ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................XXX................] duration: 511.923039ms FAIL (2025-09-23 19:01:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 511.954147ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................XXX...............] duration: 553.467116ms FAIL (2025-09-23 19:00:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 553.494026ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.X...............] duration: 389.982077ms FAIL (2025-09-23 19:00:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 390.00455ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................XXX..............] duration: 426.023346ms FAIL (2025-09-23 19:00:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 426.050257ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.X..............] duration: 368.791639ms FAIL (2025-09-23 18:59:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 368.811667ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................XXX.............] duration: 414.415176ms FAIL (2025-09-23 18:59:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 414.457045ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.X.............] duration: 439.032895ms FAIL (2025-09-23 18:59:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 439.05691ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................XXX............] duration: 384.607499ms FAIL (2025-09-23 18:58:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 384.62358ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.X............] duration: 406.464326ms FAIL (2025-09-23 18:58:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 406.488672ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................XXX...........] duration: 501.386919ms FAIL (2025-09-23 18:58:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 501.40839ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.X...........] duration: 679.356541ms FAIL (2025-09-23 18:57:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 679.379624ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.X..........] duration: 416.196759ms FAIL (2025-09-23 18:57:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 416.219832ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................XXX..........] duration: 432.818189ms FAIL (2025-09-23 18:57:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 432.841705ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.X.........] duration: 444.432501ms FAIL (2025-09-23 18:56:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 444.457989ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................XXX.........] duration: 518.77637ms FAIL (2025-09-23 18:56:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 518.804251ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.X........] duration: 586.642311ms FAIL (2025-09-23 18:55:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 586.687175ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................XXX........] duration: 699.06371ms FAIL (2025-09-23 18:55:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 699.096943ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration: 460.807549ms FAIL (2025-09-23 18:55:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 460.832215ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................XXX.......] duration: 393.071494ms FAIL (2025-09-23 18:55:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 393.089417ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.X......] duration: 461.060404ms FAIL (2025-09-23 18:55:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 461.08513ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................XXX......] duration: 615.392026ms FAIL (2025-09-23 18:54:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 615.408948ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration: 740.131807ms FAIL (2025-09-23 18:54:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 740.159689ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................XXX.....] duration: 375.29837ms FAIL (2025-09-23 18:53:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 375.3196ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration: 505.467953ms FAIL (2025-09-23 18:53:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 505.491226ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................XXX....] duration: 483.714046ms FAIL (2025-09-23 18:53:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 483.736808ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................XXX...] duration: 631.499374ms FAIL (2025-09-23 18:52:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 631.524592ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 368.233441ms FAIL (2025-09-23 18:52:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 368.263517ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XXX..] duration: 489.540643ms FAIL (2025-09-23 18:51:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 489.556875ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 675.17596ms FAIL (2025-09-23 18:51:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 675.197711ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XXX.] duration: 24.888172086s FAIL (2025-09-23 18:51:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.888184319s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.000408798s FAIL (2025-09-23 18:50:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000429437s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X.X.] duration: 7.477807461s FAIL (2025-09-23 18:50:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.477831085s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)