Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 248.992645ms FAIL (2025-08-11 22:04:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 249.017011ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...........X..........] duration: 815.475565ms FAIL (2025-08-11 22:03:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 815.498638ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [..........................XXX...............................] duration: 3.888573683s FAIL (2025-08-11 22:03:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 3.888597967s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 17.64524ms FAIL (2025-08-11 22:03:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 17.668042ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [...........................XXX..............................] duration: 2.377972375s FAIL (2025-08-11 22:03:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.377998535s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...........X.........] duration: 480.055951ms FAIL (2025-08-11 22:02:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 480.078324ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [............................XXX.............................] duration: 2.381405926s FAIL (2025-08-11 22:02:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.381433658s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 17.586701ms FAIL (2025-08-11 22:02:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:02:19.161883532 (76h35m59.893151193s) [upload=79665]: Starting upload Aug 11 22:02:19.161956230 (76h35m59.89322387s) [upload=79665]: Done starting upload Aug 11 22:02:19.162021422 (76h35m59.893289073s) [upload=79665]: Uploading segments... Aug 11 22:02:19.162062169 (76h35m59.893329839s) [upload=79665]: Got next segment. Inline: true Aug 11 22:02:19.162067379 (76h35m59.893335028s) [upload=79665]: Waiting for error group managing segments... Aug 11 22:02:19.162111532 (76h35m59.893379181s) [upload=79665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:02:19.162116450 (76h35m59.893384101s) [upload=79665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:02:19.162120057 (76h35m59.893387708s) [upload=79665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:02:19.164232361 (76h35m59.895500001s) [upload=79665]: Done uploading segments... Aug 11 22:02:19.164244493 (76h35m59.895512134s) [upload=79665]: Upload finished. err: metaclient: Too Many Requests Duration:* 17.607822ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...........X........] duration: 727.70616ms FAIL (2025-08-11 22:02:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 727.72704ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [.............................XXX............................] duration: 2.442687796s FAIL (2025-08-11 22:02:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.442711761s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X...........X.......] duration: 590.735693ms FAIL (2025-08-11 22:01:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 590.767813ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [..............................XXX...........................] duration: 4.041479815s FAIL (2025-08-11 22:01:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 4.041503541s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X...........X......] duration: 1.796087313s FAIL (2025-08-11 22:01:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 1.796108131s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X...........X.....] duration: 748.152442ms FAIL (2025-08-11 22:01:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 748.182969ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [...............................XXX..........................] duration: 4.310967764s FAIL (2025-08-11 22:00:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 4.310993451s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X...........X....] duration: 987.63671ms FAIL (2025-08-11 22:00:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 987.668931ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [................................XXX.........................] duration: 7.704677898s FAIL (2025-08-11 22:00:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 7.704700922s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X...........X...] duration: 606.656483ms FAIL (2025-08-11 22:00:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 606.680439ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X...........X..] duration: 823.947892ms FAIL (2025-08-11 21:59:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 823.975575ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [.................................XXX........................] duration: 2.306851588s FAIL (2025-08-11 21:59:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.306882185s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X...........X.] duration: 812.183975ms FAIL (2025-08-11 21:59:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 812.203042ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X...........X] duration: 16.906793ms FAIL (2025-08-11 21:58:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 16.923734ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [..................................XXX.......................] duration: 2.232092688s FAIL (2025-08-11 21:58:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.232108106s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 911.636953ms FAIL (2025-08-11 21:58:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:50:28.753915051 (76h24m9.485182681s) [upload=79465]: Starting upload Aug 11 21:50:28.753953273 (76h24m9.485220903s) [upload=79465]: Done starting upload Aug 11 21:50:28.754031010 (76h24m9.48529863s) [upload=79465]: Uploading segments... Aug 11 21:50:28.754295096 (76h24m9.485562726s) [upload=79465]: Got next segment. Inline: false Aug 11 21:50:28.754418448 (76h24m9.485686068s) [upload=79465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:50:28.754425873 (76h24m9.485693503s) [upload=79465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:50:28.756928379 (76h24m9.488196039s) [upload=79465]: Next segment err: metaclient: Too Many Requests Aug 11 21:50:28.756964006 (76h24m9.488231625s) [upload=79465]: Done uploading segments... Aug 11 21:50:28.757006285 (76h24m9.488273915s) [upload=79465]: Upload finished. err: metaclient: Too Many Requests Duration: 911.658534ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [...................................XXX......................] duration: 2.208991995s FAIL (2025-08-11 21:57:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.209018736s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 706.370059ms FAIL (2025-08-11 21:57:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:50:28.753915051 (76h24m9.485182681s) [upload=79465]: Starting upload Aug 11 21:50:28.753953273 (76h24m9.485220903s) [upload=79465]: Done starting upload Aug 11 21:50:28.754031010 (76h24m9.48529863s) [upload=79465]: Uploading segments... Aug 11 21:50:28.754295096 (76h24m9.485562726s) [upload=79465]: Got next segment. Inline: false Aug 11 21:50:28.754418448 (76h24m9.485686068s) [upload=79465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:50:28.754425873 (76h24m9.485693503s) [upload=79465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:50:28.756928379 (76h24m9.488196039s) [upload=79465]: Next segment err: metaclient: Too Many Requests Aug 11 21:50:28.756964006 (76h24m9.488231625s) [upload=79465]: Done uploading segments... Aug 11 21:50:28.757006285 (76h24m9.488273915s) [upload=79465]: Upload finished. err: metaclient: Too Many Requests Duration: 706.449528ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [....................................XXX.....................] duration: 2.216420979s FAIL (2025-08-11 21:56:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.216449693s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 1.264261171s FAIL (2025-08-11 21:56:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:50:28.753915051 (76h24m9.485182681s) [upload=79465]: Starting upload Aug 11 21:50:28.753953273 (76h24m9.485220903s) [upload=79465]: Done starting upload Aug 11 21:50:28.754031010 (76h24m9.48529863s) [upload=79465]: Uploading segments... Aug 11 21:50:28.754295096 (76h24m9.485562726s) [upload=79465]: Got next segment. Inline: false Aug 11 21:50:28.754418448 (76h24m9.485686068s) [upload=79465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:50:28.754425873 (76h24m9.485693503s) [upload=79465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:50:28.756928379 (76h24m9.488196039s) [upload=79465]: Next segment err: metaclient: Too Many Requests Aug 11 21:50:28.756964006 (76h24m9.488231625s) [upload=79465]: Done uploading segments... Aug 11 21:50:28.757006285 (76h24m9.488273915s) [upload=79465]: Upload finished. err: metaclient: Too Many Requests Duration: 1.26428209s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [.....................................XXX....................] duration: 2.247740871s FAIL (2025-08-11 21:56:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.247760668s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 801.032038ms FAIL (2025-08-11 21:56:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:50:28.753915051 (76h24m9.485182681s) [upload=79465]: Starting upload Aug 11 21:50:28.753953273 (76h24m9.485220903s) [upload=79465]: Done starting upload Aug 11 21:50:28.754031010 (76h24m9.48529863s) [upload=79465]: Uploading segments... Aug 11 21:50:28.754295096 (76h24m9.485562726s) [upload=79465]: Got next segment. Inline: false Aug 11 21:50:28.754418448 (76h24m9.485686068s) [upload=79465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:50:28.754425873 (76h24m9.485693503s) [upload=79465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:50:28.756928379 (76h24m9.488196039s) [upload=79465]: Next segment err: metaclient: Too Many Requests Aug 11 21:50:28.756964006 (76h24m9.488231625s) [upload=79465]: Done uploading segments... Aug 11 21:50:28.757006285 (76h24m9.488273915s) [upload=79465]: Upload finished. err: metaclient: Too Many Requests Duration: 801.071702ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [......................................XXX...................] duration: 2.254779421s FAIL (2025-08-11 21:55:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.25481087s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 820.704703ms FAIL (2025-08-11 21:55:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:50:28.753915051 (76h24m9.485182681s) [upload=79465]: Starting upload Aug 11 21:50:28.753953273 (76h24m9.485220903s) [upload=79465]: Done starting upload Aug 11 21:50:28.754031010 (76h24m9.48529863s) [upload=79465]: Uploading segments... Aug 11 21:50:28.754295096 (76h24m9.485562726s) [upload=79465]: Got next segment. Inline: false Aug 11 21:50:28.754418448 (76h24m9.485686068s) [upload=79465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:50:28.754425873 (76h24m9.485693503s) [upload=79465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:50:28.756928379 (76h24m9.488196039s) [upload=79465]: Next segment err: metaclient: Too Many Requests Aug 11 21:50:28.756964006 (76h24m9.488231625s) [upload=79465]: Done uploading segments... Aug 11 21:50:28.757006285 (76h24m9.488273915s) [upload=79465]: Upload finished. err: metaclient: Too Many Requests Duration: 820.732144ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [.......................................XXX..................] duration: 2.238155219s FAIL (2025-08-11 21:54:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.238185455s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [........................................XXX.................] duration: 2.25897381s FAIL (2025-08-11 21:54:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.25910667s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 537.466039ms FAIL (2025-08-11 21:54:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:50:28.753915051 (76h24m9.485182681s) [upload=79465]: Starting upload Aug 11 21:50:28.753953273 (76h24m9.485220903s) [upload=79465]: Done starting upload Aug 11 21:50:28.754031010 (76h24m9.48529863s) [upload=79465]: Uploading segments... Aug 11 21:50:28.754295096 (76h24m9.485562726s) [upload=79465]: Got next segment. Inline: false Aug 11 21:50:28.754418448 (76h24m9.485686068s) [upload=79465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:50:28.754425873 (76h24m9.485693503s) [upload=79465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:50:28.756928379 (76h24m9.488196039s) [upload=79465]: Next segment err: metaclient: Too Many Requests Aug 11 21:50:28.756964006 (76h24m9.488231625s) [upload=79465]: Done uploading segments... Aug 11 21:50:28.757006285 (76h24m9.488273915s) [upload=79465]: Upload finished. err: metaclient: Too Many Requests Duration: 537.492307ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.632548459s FAIL (2025-08-11 21:54:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:50:28.753915051 (76h24m9.485182681s) [upload=79465]: Starting upload Aug 11 21:50:28.753953273 (76h24m9.485220903s) [upload=79465]: Done starting upload Aug 11 21:50:28.754031010 (76h24m9.48529863s) [upload=79465]: Uploading segments... Aug 11 21:50:28.754295096 (76h24m9.485562726s) [upload=79465]: Got next segment. Inline: false Aug 11 21:50:28.754418448 (76h24m9.485686068s) [upload=79465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:50:28.754425873 (76h24m9.485693503s) [upload=79465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:50:28.756928379 (76h24m9.488196039s) [upload=79465]: Next segment err: metaclient: Too Many Requests Aug 11 21:50:28.756964006 (76h24m9.488231625s) [upload=79465]: Done uploading segments... Aug 11 21:50:28.757006285 (76h24m9.488273915s) [upload=79465]: Upload finished. err: metaclient: Too Many Requests Duration: 1.632571832s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [.........................................XXX................] duration: 2.369643658s FAIL (2025-08-11 21:54:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.369663595s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.100799169s FAIL (2025-08-11 21:53:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:50:28.753915051 (76h24m9.485182681s) [upload=79465]: Starting upload Aug 11 21:50:28.753953273 (76h24m9.485220903s) [upload=79465]: Done starting upload Aug 11 21:50:28.754031010 (76h24m9.48529863s) [upload=79465]: Uploading segments... Aug 11 21:50:28.754295096 (76h24m9.485562726s) [upload=79465]: Got next segment. Inline: false Aug 11 21:50:28.754418448 (76h24m9.485686068s) [upload=79465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:50:28.754425873 (76h24m9.485693503s) [upload=79465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:50:28.756928379 (76h24m9.488196039s) [upload=79465]: Next segment err: metaclient: Too Many Requests Aug 11 21:50:28.756964006 (76h24m9.488231625s) [upload=79465]: Done uploading segments... Aug 11 21:50:28.757006285 (76h24m9.488273915s) [upload=79465]: Upload finished. err: metaclient: Too Many Requests Duration: 1.100821903s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [..........................................XXX...............] duration: 2.231302618s FAIL (2025-08-11 21:52:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.231324057s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [...........................................XXX..............] duration: 2.280602164s FAIL (2025-08-11 21:52:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.280685151s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 966.41713ms FAIL (2025-08-11 21:52:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:50:28.753915051 (76h24m9.485182681s) [upload=79465]: Starting upload Aug 11 21:50:28.753953273 (76h24m9.485220903s) [upload=79465]: Done starting upload Aug 11 21:50:28.754031010 (76h24m9.48529863s) [upload=79465]: Uploading segments... Aug 11 21:50:28.754295096 (76h24m9.485562726s) [upload=79465]: Got next segment. Inline: false Aug 11 21:50:28.754418448 (76h24m9.485686068s) [upload=79465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:50:28.754425873 (76h24m9.485693503s) [upload=79465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:50:28.756928379 (76h24m9.488196039s) [upload=79465]: Next segment err: metaclient: Too Many Requests Aug 11 21:50:28.756964006 (76h24m9.488231625s) [upload=79465]: Done uploading segments... Aug 11 21:50:28.757006285 (76h24m9.488273915s) [upload=79465]: Upload finished. err: metaclient: Too Many Requests Duration: 966.442237ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [............................................XXX.............] duration: 2.300349234s FAIL (2025-08-11 21:52:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.300370063s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 963.682425ms FAIL (2025-08-11 21:51:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:50:28.753915051 (76h24m9.485182681s) [upload=79465]: Starting upload Aug 11 21:50:28.753953273 (76h24m9.485220903s) [upload=79465]: Done starting upload Aug 11 21:50:28.754031010 (76h24m9.48529863s) [upload=79465]: Uploading segments... Aug 11 21:50:28.754295096 (76h24m9.485562726s) [upload=79465]: Got next segment. Inline: false Aug 11 21:50:28.754418448 (76h24m9.485686068s) [upload=79465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:50:28.754425873 (76h24m9.485693503s) [upload=79465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:50:28.756928379 (76h24m9.488196039s) [upload=79465]: Next segment err: metaclient: Too Many Requests Aug 11 21:50:28.756964006 (76h24m9.488231625s) [upload=79465]: Done uploading segments... Aug 11 21:50:28.757006285 (76h24m9.488273915s) [upload=79465]: Upload finished. err: metaclient: Too Many Requests Duration: 963.708083ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [.............................................XXX............] duration: 2.288102782s FAIL (2025-08-11 21:51:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.288127498s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 417.49412ms FAIL (2025-08-11 21:51:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:50:28.753915051 (76h24m9.485182681s) [upload=79465]: Starting upload Aug 11 21:50:28.753953273 (76h24m9.485220903s) [upload=79465]: Done starting upload Aug 11 21:50:28.754031010 (76h24m9.48529863s) [upload=79465]: Uploading segments... Aug 11 21:50:28.754295096 (76h24m9.485562726s) [upload=79465]: Got next segment. Inline: false Aug 11 21:50:28.754418448 (76h24m9.485686068s) [upload=79465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:50:28.754425873 (76h24m9.485693503s) [upload=79465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:50:28.756928379 (76h24m9.488196039s) [upload=79465]: Next segment err: metaclient: Too Many Requests Aug 11 21:50:28.756964006 (76h24m9.488231625s) [upload=79465]: Done uploading segments... Aug 11 21:50:28.757006285 (76h24m9.488273915s) [upload=79465]: Upload finished. err: metaclient: Too Many Requests Duration: 417.508527ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [..............................................XXX...........] duration: 2.38573044s FAIL (2025-08-11 21:50:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.385754585s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 15.130727ms FAIL (2025-08-11 21:50:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:50:28.753915051 (76h24m9.485182681s) [upload=79465]: Starting upload Aug 11 21:50:28.753953273 (76h24m9.485220903s) [upload=79465]: Done starting upload Aug 11 21:50:28.754031010 (76h24m9.48529863s) [upload=79465]: Uploading segments... Aug 11 21:50:28.754295096 (76h24m9.485562726s) [upload=79465]: Got next segment. Inline: false Aug 11 21:50:28.754418448 (76h24m9.485686068s) [upload=79465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:50:28.754425873 (76h24m9.485693503s) [upload=79465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:50:28.756928379 (76h24m9.488196039s) [upload=79465]: Next segment err: metaclient: Too Many Requests Aug 11 21:50:28.756964006 (76h24m9.488231625s) [upload=79465]: Done uploading segments... Aug 11 21:50:28.757006285 (76h24m9.488273915s) [upload=79465]: Upload finished. err: metaclient: Too Many Requests Duration: 15.152538ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [...............................................XXX..........] duration: 2.230396995s FAIL (2025-08-11 21:49:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.230420749s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [................................................XXX.........] duration: 2.310584722s FAIL (2025-08-11 21:48:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.310606864s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [.................................................XXX........] duration: 2.254682848s FAIL (2025-08-11 21:48:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.254703837s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)