Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [........XXX.................................................] duration: 2.288909312s FAIL (2025-08-11 22:17:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.288934971s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 230.369232ms FAIL (2025-08-11 22:16:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 230.411171ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...........X........................] duration: 537.79589ms FAIL (2025-08-11 22:16:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 537.819084ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [.........XXX................................................] duration: 2.216400619s FAIL (2025-08-11 22:16:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.216423853s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 517.985135ms FAIL (2025-08-11 22:16:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 518.005162ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X...........X.......................] duration: 736.146762ms FAIL (2025-08-11 22:16:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 736.171449ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 175.674586ms FAIL (2025-08-11 22:15:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 175.700965ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [..........XXX...............................................] duration: 2.324021907s FAIL (2025-08-11 22:15:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.324048717s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 266.761252ms FAIL (2025-08-11 22:15:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 266.781861ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...........X......................] duration: 904.828324ms FAIL (2025-08-11 22:14:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 904.855826ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [...........XXX..............................................] duration: 2.515229114s FAIL (2025-08-11 22:14:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.515251195s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 263.169725ms FAIL (2025-08-11 22:14:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 263.189783ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...........X.....................] duration: 614.35506ms FAIL (2025-08-11 22:13:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 614.391408ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [............XXX.............................................] duration: 2.376777393s FAIL (2025-08-11 22:13:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.376811588s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 174.981063ms FAIL (2025-08-11 22:13:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 175.003585ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...........X....................] duration: 670.443817ms FAIL (2025-08-11 22:13:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 670.471749ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [.............XXX............................................] duration: 2.368621641s FAIL (2025-08-11 22:13:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.368649183s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 383.036313ms FAIL (2025-08-11 22:12:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 383.060398ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X...........X...................] duration: 880.688395ms FAIL (2025-08-11 22:12:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 880.709947ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 282.001815ms FAIL (2025-08-11 22:11:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 282.023265ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [..............XXX...........................................] duration: 2.378105461s FAIL (2025-08-11 22:11:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.378143071s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X...........X..................] duration: 980.532905ms FAIL (2025-08-11 22:11:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 980.566316ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [...............XXX..........................................] duration: 2.587004417s FAIL (2025-08-11 22:11:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.587024315s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 233.201472ms FAIL (2025-08-11 22:11:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 233.222382ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...........X.................] duration: 633.535403ms FAIL (2025-08-11 22:10:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 633.556581ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [................XXX.........................................] duration: 2.747371808s FAIL (2025-08-11 22:10:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.74739935s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 191.67073ms FAIL (2025-08-11 22:10:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 191.696388ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [.................XXX........................................] duration: 2.270431423s FAIL (2025-08-11 22:09:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.270459686s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...........X................] duration: 777.343458ms FAIL (2025-08-11 22:09:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 777.372333ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 182.481046ms FAIL (2025-08-11 22:09:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 182.503678ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [..................XXX.......................................] duration: 3.2803529s FAIL (2025-08-11 22:09:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 3.280373519s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 214.535109ms FAIL (2025-08-11 22:08:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 214.564074ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [...................XXX......................................] duration: 3.081575027s FAIL (2025-08-11 22:08:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 3.081598031s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 240.560454ms FAIL (2025-08-11 22:08:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 240.582386ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X...........X...............] duration: 931.988547ms FAIL (2025-08-11 22:08:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 932.01732ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [....................XXX.....................................] duration: 3.047317989s FAIL (2025-08-11 22:07:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 3.047341524s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X...........X..............] duration: 562.526951ms FAIL (2025-08-11 22:07:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 562.551247ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [.....................XXX....................................] duration: 3.165719478s FAIL (2025-08-11 22:07:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 3.165738463s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 167.25712ms FAIL (2025-08-11 22:07:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 167.275885ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X...........X.............] duration: 466.37546ms FAIL (2025-08-11 22:06:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 466.39695ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [......................XXX...................................] duration: 3.43191097s FAIL (2025-08-11 22:06:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 3.431939604s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 267.191446ms FAIL (2025-08-11 22:06:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 267.21554ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [.......................XXX..................................] duration: 2.338251448s FAIL (2025-08-11 22:06:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.338283408s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [........................XXX.................................] duration: 3.520635321s FAIL (2025-08-11 22:05:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 3.520671651s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...........X............] duration: 924.291977ms FAIL (2025-08-11 22:05:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 924.31889ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 200.98985ms FAIL (2025-08-11 22:05:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 201.016722ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 158.708657ms FAIL (2025-08-11 22:05:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 158.732843ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [.........................XXX................................] duration: 2.312807821s FAIL (2025-08-11 22:04:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 16) ======LOGS====== 0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece... Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments... Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil> Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil> Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments... Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16) Duration: 2.312840312s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...........X...........] duration: 539.41671ms FAIL (2025-08-11 22:04:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests Duration: 539.443801ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 248.992645ms FAIL (2025-08-11 22:04:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests Duration:* 249.017011ms core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)