Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......XXX..................................................] duration: 434.762941ms FAIL (2025-09-23 19:28:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 434.791105ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.X.................................................] duration: 394.563483ms FAIL (2025-09-23 19:27:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 394.590674ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........XXX.................................................] duration: 455.513946ms FAIL (2025-09-23 19:27:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 455.533483ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.X................................................] duration: 430.744415ms FAIL (2025-09-23 19:27:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 430.766537ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.X...............................................] duration: 370.51681ms FAIL (2025-09-23 19:26:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 370.534342ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.X..............................................] duration: 320.85875ms FAIL (2025-09-23 19:26:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 320.879238ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........XXX................................................] duration: 389.428058ms FAIL (2025-09-23 19:26:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 389.446152ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........XXX...............................................] duration: 460.963341ms FAIL (2025-09-23 19:26:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 460.98907ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........XXX..............................................] duration: 677.896708ms FAIL (2025-09-23 19:25:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 677.91413ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.X.............................................] duration: 441.317616ms FAIL (2025-09-23 19:25:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 441.341441ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............XXX.............................................] duration: 417.931423ms FAIL (2025-09-23 19:25:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 417.949268ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.X............................................] duration: 355.042815ms FAIL (2025-09-23 19:24:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 355.072711ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.X...........................................] duration: 456.915581ms FAIL (2025-09-23 19:24:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 456.941639ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............XXX............................................] duration: 393.836332ms FAIL (2025-09-23 19:23:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 393.856029ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............XXX...........................................] duration: 425.430829ms FAIL (2025-09-23 19:23:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 425.456838ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.X..........................................] duration: 523.582015ms FAIL (2025-09-23 19:22:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 523.679209ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............XXX..........................................] duration: 512.41394ms FAIL (2025-09-23 19:22:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 512.452463ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.X.........................................] duration: 423.449285ms FAIL (2025-09-23 19:21:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 423.471015ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................XXX.........................................] duration: 393.816893ms FAIL (2025-09-23 19:21:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 393.836109ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.X........................................] duration: 366.627221ms FAIL (2025-09-23 19:20:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 366.653049ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.X.......................................] duration: 636.231844ms FAIL (2025-09-23 19:20:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 636.255299ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................XXX........................................] duration: 420.795775ms FAIL (2025-09-23 19:20:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 420.813888ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.X......................................] duration: 472.391567ms FAIL (2025-09-23 19:19:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 472.415351ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................XXX.......................................] duration: 827.398263ms FAIL (2025-09-23 19:19:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 827.422087ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.X.....................................] duration: 419.884405ms FAIL (2025-09-23 19:18:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 419.912347ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................XXX......................................] duration: 453.875351ms FAIL (2025-09-23 19:18:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 453.897933ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.X....................................] duration: 411.581523ms FAIL (2025-09-23 19:18:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 411.601781ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................XXX.....................................] duration: 393.596529ms FAIL (2025-09-23 19:18:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 393.615224ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.X...................................] duration: 428.306803ms FAIL (2025-09-23 19:17:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 428.334285ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................XXX....................................] duration: 423.07882ms FAIL (2025-09-23 19:17:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 423.126519ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.X..................................] duration: 362.399773ms FAIL (2025-09-23 19:16:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 362.418639ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................XXX...................................] duration: 401.227074ms FAIL (2025-09-23 19:16:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 401.249858ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.X.................................] duration: 395.167178ms FAIL (2025-09-23 19:15:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 395.201693ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................XXX..................................] duration: 340.913275ms FAIL (2025-09-23 19:15:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 340.932411ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.X................................] duration: 506.600194ms FAIL (2025-09-23 19:14:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 506.633707ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................XXX.................................] duration: 404.147962ms FAIL (2025-09-23 19:14:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 404.165194ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.X...............................] duration: 392.907861ms FAIL (2025-09-23 19:14:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 392.94496ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.X..............................] duration: 425.874292ms FAIL (2025-09-23 19:13:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 425.892476ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................XXX................................] duration: 336.398559ms FAIL (2025-09-23 19:13:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 336.418046ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.X.............................] duration: 514.486224ms FAIL (2025-09-23 19:12:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 514.511643ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................XXX...............................] duration: 384.273878ms FAIL (2025-09-23 19:12:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 384.296782ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.X............................] duration: 406.21557ms FAIL (2025-09-23 19:11:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 406.235116ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................XXX..............................] duration: 434.204688ms FAIL (2025-09-23 19:11:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 434.256665ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.X...........................] duration: 519.808839ms FAIL (2025-09-23 19:10:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 519.836923ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................XXX.............................] duration: 715.834361ms FAIL (2025-09-23 19:10:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 715.851914ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................XXX............................] duration: 370.392602ms FAIL (2025-09-23 19:10:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 370.416697ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.X..........................] duration: 351.948662ms FAIL (2025-09-23 19:09:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 351.970232ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................XXX...........................] duration: 394.94397ms FAIL (2025-09-23 19:09:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 394.960682ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.X.........................] duration: 527.846642ms FAIL (2025-09-23 19:09:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 527.86638ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.X........................] duration: 337.208532ms FAIL (2025-09-23 19:08:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 337.232717ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)