Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................XXXX.......................] duration: 1.229893842s FAIL (2025-08-17 04:40:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 04:23:05.076746715 (202h56m45.808014345s) [upload=211167]: Starting upload Aug 17 04:23:05.076819060 (202h56m45.80808669s) [upload=211167]: Done starting upload Aug 17 04:23:05.076858426 (202h56m45.808126045s) [upload=211167]: Uploading segments... Aug 17 04:23:05.077035118 (202h56m45.808302748s) [upload=211167]: Got next segment. Inline: false Aug 17 04:23:05.077065765 (202h56m45.808333395s) [upload=211167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:05.077069302 (202h56m45.808336922s) [upload=211167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:05.149595571 (202h56m45.880863191s) [seg_pos={0 0}, upload=211167]: Begin upload segment... Aug 17 04:23:05.149634344 (202h56m45.880901974s) [seg_pos={0 0}, upload=211167]: Done begin upload segment. Aug 17 04:23:05.149659451 (202h56m45.880927081s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.149699086 (202h56m45.880966716s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150081695 (202h56m45.881349325s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150110339 (202h56m45.88137799s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150390978 (202h56m45.881658608s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150446372 (202h56m45.881714002s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150693598 (202h56m45.881961228s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150755163 (202h56m45.882022793s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150983263 (202h56m45.882250893s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151079885 (202h56m45.882347515s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151274822 (202h56m45.882542452s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151365532 (202h56m45.882633163s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151584305 (202h56m45.882851935s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151603451 (202h56m45.882871081s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.152583125 (202h56m45.883850755s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.153130696 (202h56m45.884398326s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154141749 (202h56m45.88540938s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154561108 (202h56m45.885828738s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.222341082 (202h56m45.953608762s) [upload=211167]: Next returned nil segment Aug 17 04:23:05.222354026 (202h56m45.953621646s) [upload=211167]: Waiting for error group managing segments... Aug 17 04:23:05.230308020 (202h56m45.96157565s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.230754350 (202h56m45.96202198s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231421757 (202h56m45.962689387s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231643444 (202h56m45.962911074s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.296297613 (202h56m46.027565233s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.346576712 (202h56m46.077844332s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.494438107 (202h56m46.225705727s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.563775022 (202h56m46.295042642s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.674589117 (202h56m46.405856747s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.747386525 (202h56m46.478654145s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:24:05.061032967 (202h57m45.792300617s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:104.36.82.60:4351): protocol: rpc: dial tcp 104.36.82.60:4351: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061097619 (202h57m45.79236527s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: rpc: dial tcp 104.36.82.38:4696: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061148564 (202h57m45.792416215s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: rpc: dial tcp 104.36.82.17:4333: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061255446 (202h57m45.792523106s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12Nh9nHAnKWgu1ggCAZ7WcUadbvsPqAgbqzGrBnSaDMqDhquQ4R, address:104.36.82.22:4514): protocol: rpc: dial tcp 104.36.82.22:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061322663 (202h57m45.792590312s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, address:104.36.82.23:4446): protocol: write tcp 5.161.89.252:28480->104.36.82.23:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061363470 (202h57m45.79263112s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: rpc: dial tcp 104.36.82.2:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061387064 (202h57m45.792654714s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u, address:104.36.82.5:4864): protocol: rpc: dial tcp 104.36.82.5:4864: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061424123 (202h57m45.792691784s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA, address:104.36.82.23:4461): protocol: write tcp 5.161.89.252:18768->104.36.82.23:4461: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061453258 (202h57m45.792720908s) [seg_pos={0 0}, upload=211167]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:24:05.085107047 (202h57m45.816374697s) [upload=211167]: Done uploading segments... Aug 17 04:24:05.085125321 (202h57m45.816392971s) [upload=211167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1.229913549s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.503231543s FAIL (2025-08-17 04:40:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.503245809s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................XXXXXX.......................] duration: 564.148846ms FAIL (2025-08-17 04:40:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:07.365016247 (202h56m48.096283887s) [upload=211168]: Starting upload Aug 17 04:23:07.365113880 (202h56m48.09638152s) [upload=211168]: Done starting upload Aug 17 04:23:07.365185295 (202h56m48.096452915s) [upload=211168]: Uploading segments... Aug 17 04:23:07.365444052 (202h56m48.096711693s) [upload=211168]: Got next segment. Inline: false Aug 17 04:23:07.365496200 (202h56m48.09676384s) [upload=211168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:07.365506590 (202h56m48.096774229s) [upload=211168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:07.391532593 (202h56m48.122800213s) [seg_pos={0 0}, upload=211168]: Begin upload segment... Aug 17 04:23:07.391561247 (202h56m48.122828867s) [seg_pos={0 0}, upload=211168]: Done begin upload segment. Aug 17 04:23:07.391580673 (202h56m48.122848294s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391635377 (202h56m48.122903017s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391895125 (202h56m48.123162745s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392097867 (202h56m48.123365508s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392117784 (202h56m48.123385405s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392313993 (202h56m48.123581614s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392395668 (202h56m48.123663308s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392493893 (202h56m48.123761513s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392681466 (202h56m48.123949106s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392702645 (202h56m48.123970275s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392977794 (202h56m48.124245414s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392988844 (202h56m48.124256475s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393185134 (202h56m48.124452754s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393273310 (202h56m48.12454094s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393400339 (202h56m48.124667959s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393572042 (202h56m48.124839683s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.461635871 (202h56m48.192903501s) [upload=211168]: Next returned nil segment Aug 17 04:23:07.461646972 (202h56m48.192914602s) [upload=211168]: Waiting for error group managing segments... Aug 17 04:23:07.469880881 (202h56m48.201148501s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470249144 (202h56m48.201516775s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470491772 (202h56m48.201759402s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470580539 (202h56m48.201848169s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470934566 (202h56m48.202202196s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471147235 (202h56m48.202414855s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471257373 (202h56m48.202525003s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471574630 (202h56m48.20284225s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471656044 (202h56m48.202923674s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.472781833 (202h56m48.204049453s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:37.353223355 (202h57m18.084491014s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353345955 (202h57m18.084613605s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353365382 (202h57m18.084633042s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353416519 (202h57m18.084684169s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353477142 (202h57m18.084744793s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 5.161.89.252:26914->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353487252 (202h57m18.084754901s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353530393 (202h57m18.084798093s) [seg_pos={0 0}, upload=211168]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:37.392615539 (202h57m18.123883199s) [upload=211168]: Done uploading segments... Aug 17 04:23:37.392649854 (202h57m18.123917504s) [upload=211168]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 564.175867ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................XXXX......................] duration: 592.941591ms FAIL (2025-08-17 04:40:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 04:23:05.076746715 (202h56m45.808014345s) [upload=211167]: Starting upload Aug 17 04:23:05.076819060 (202h56m45.80808669s) [upload=211167]: Done starting upload Aug 17 04:23:05.076858426 (202h56m45.808126045s) [upload=211167]: Uploading segments... Aug 17 04:23:05.077035118 (202h56m45.808302748s) [upload=211167]: Got next segment. Inline: false Aug 17 04:23:05.077065765 (202h56m45.808333395s) [upload=211167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:05.077069302 (202h56m45.808336922s) [upload=211167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:05.149595571 (202h56m45.880863191s) [seg_pos={0 0}, upload=211167]: Begin upload segment... Aug 17 04:23:05.149634344 (202h56m45.880901974s) [seg_pos={0 0}, upload=211167]: Done begin upload segment. Aug 17 04:23:05.149659451 (202h56m45.880927081s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.149699086 (202h56m45.880966716s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150081695 (202h56m45.881349325s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150110339 (202h56m45.88137799s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150390978 (202h56m45.881658608s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150446372 (202h56m45.881714002s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150693598 (202h56m45.881961228s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150755163 (202h56m45.882022793s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150983263 (202h56m45.882250893s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151079885 (202h56m45.882347515s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151274822 (202h56m45.882542452s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151365532 (202h56m45.882633163s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151584305 (202h56m45.882851935s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151603451 (202h56m45.882871081s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.152583125 (202h56m45.883850755s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.153130696 (202h56m45.884398326s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154141749 (202h56m45.88540938s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154561108 (202h56m45.885828738s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.222341082 (202h56m45.953608762s) [upload=211167]: Next returned nil segment Aug 17 04:23:05.222354026 (202h56m45.953621646s) [upload=211167]: Waiting for error group managing segments... Aug 17 04:23:05.230308020 (202h56m45.96157565s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.230754350 (202h56m45.96202198s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231421757 (202h56m45.962689387s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231643444 (202h56m45.962911074s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.296297613 (202h56m46.027565233s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.346576712 (202h56m46.077844332s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.494438107 (202h56m46.225705727s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.563775022 (202h56m46.295042642s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.674589117 (202h56m46.405856747s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.747386525 (202h56m46.478654145s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:24:05.061032967 (202h57m45.792300617s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:104.36.82.60:4351): protocol: rpc: dial tcp 104.36.82.60:4351: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061097619 (202h57m45.79236527s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: rpc: dial tcp 104.36.82.38:4696: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061148564 (202h57m45.792416215s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: rpc: dial tcp 104.36.82.17:4333: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061255446 (202h57m45.792523106s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12Nh9nHAnKWgu1ggCAZ7WcUadbvsPqAgbqzGrBnSaDMqDhquQ4R, address:104.36.82.22:4514): protocol: rpc: dial tcp 104.36.82.22:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061322663 (202h57m45.792590312s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, address:104.36.82.23:4446): protocol: write tcp 5.161.89.252:28480->104.36.82.23:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061363470 (202h57m45.79263112s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: rpc: dial tcp 104.36.82.2:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061387064 (202h57m45.792654714s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u, address:104.36.82.5:4864): protocol: rpc: dial tcp 104.36.82.5:4864: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061424123 (202h57m45.792691784s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA, address:104.36.82.23:4461): protocol: write tcp 5.161.89.252:18768->104.36.82.23:4461: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061453258 (202h57m45.792720908s) [seg_pos={0 0}, upload=211167]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:24:05.085107047 (202h57m45.816374697s) [upload=211167]: Done uploading segments... Aug 17 04:24:05.085125321 (202h57m45.816392971s) [upload=211167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 592.965075ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................XXXX....................] duration: 394.03667ms FAIL (2025-08-17 04:40:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== se Aug 17 04:23:28.620840633 (202h57m9.352108284s) [upload=211175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:28.620845953 (202h57m9.352113603s) [upload=211175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:28.646835158 (202h57m9.378102858s) [seg_pos={0 0}, upload=211175]: Begin upload segment... Aug 17 04:23:28.646936819 (202h57m9.378204468s) [seg_pos={0 0}, upload=211175]: Done begin upload segment. Aug 17 04:23:28.646989559 (202h57m9.378257218s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647051715 (202h57m9.378319375s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647662535 (202h57m9.378930186s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647827836 (202h57m9.379095487s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648134293 (202h57m9.379401954s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648314913 (202h57m9.379582573s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648543414 (202h57m9.379811063s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648834572 (202h57m9.380102222s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648992249 (202h57m9.380259908s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649335063 (202h57m9.380602713s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649507127 (202h57m9.380774777s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649830226 (202h57m9.381097876s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650003341 (202h57m9.381270992s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650256469 (202h57m9.381524128s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650434553 (202h57m9.381702213s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650709621 (202h57m9.381977282s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650857891 (202h57m9.38212554s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.651270246 (202h57m9.382537897s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.768946979 (202h57m9.500214629s) [upload=211175]: Next returned nil segment Aug 17 04:23:28.768974191 (202h57m9.500241841s) [upload=211175]: Waiting for error group managing segments... Aug 17 04:23:28.777572686 (202h57m9.508840347s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.778945461 (202h57m9.510213112s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.785896566 (202h57m9.517164206s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.793497595 (202h57m9.524765245s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.796220951 (202h57m9.527488592s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.799941495 (202h57m9.531209135s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.801171361 (202h57m9.532439001s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.802452863 (202h57m9.533720513s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.822657288 (202h57m9.553924938s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.874167434 (202h57m9.605435084s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.100609905 (202h57m9.831877555s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.216981541 (202h57m9.948249171s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:24:28.606052339 (202h58m9.337319999s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: rpc: dial tcp 104.36.82.44:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606141437 (202h58m9.337409087s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1AowWwVvxzmnW5qwbqbMBVbvNoigffjAN51LQp8e6XA9CZkeXL, address:104.36.82.3:4790): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606261834 (202h58m9.337529484s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:58906->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606338818 (202h58m9.337606469s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:104.36.82.23:4455): protocol: write tcp 5.161.89.252:44290->104.36.82.23:4455: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606369045 (202h58m9.337636696s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: rpc: dial tcp 104.36.82.60:4365: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606443665 (202h58m9.337711316s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, address:104.36.82.49:4003): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606480044 (202h58m9.337747704s) [seg_pos={0 0}, upload=211175]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 04:24:28.621101719 (202h58m9.352369369s) [upload=211175]: Done uploading segments... Aug 17 04:24:28.621117609 (202h58m9.35238526s) [upload=211175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 394.054733ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................XXXXXX......................] duration: 347.54036ms FAIL (2025-08-17 04:39:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:07.365016247 (202h56m48.096283887s) [upload=211168]: Starting upload Aug 17 04:23:07.365113880 (202h56m48.09638152s) [upload=211168]: Done starting upload Aug 17 04:23:07.365185295 (202h56m48.096452915s) [upload=211168]: Uploading segments... Aug 17 04:23:07.365444052 (202h56m48.096711693s) [upload=211168]: Got next segment. Inline: false Aug 17 04:23:07.365496200 (202h56m48.09676384s) [upload=211168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:07.365506590 (202h56m48.096774229s) [upload=211168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:07.391532593 (202h56m48.122800213s) [seg_pos={0 0}, upload=211168]: Begin upload segment... Aug 17 04:23:07.391561247 (202h56m48.122828867s) [seg_pos={0 0}, upload=211168]: Done begin upload segment. Aug 17 04:23:07.391580673 (202h56m48.122848294s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391635377 (202h56m48.122903017s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391895125 (202h56m48.123162745s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392097867 (202h56m48.123365508s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392117784 (202h56m48.123385405s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392313993 (202h56m48.123581614s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392395668 (202h56m48.123663308s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392493893 (202h56m48.123761513s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392681466 (202h56m48.123949106s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392702645 (202h56m48.123970275s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392977794 (202h56m48.124245414s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392988844 (202h56m48.124256475s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393185134 (202h56m48.124452754s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393273310 (202h56m48.12454094s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393400339 (202h56m48.124667959s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393572042 (202h56m48.124839683s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.461635871 (202h56m48.192903501s) [upload=211168]: Next returned nil segment Aug 17 04:23:07.461646972 (202h56m48.192914602s) [upload=211168]: Waiting for error group managing segments... Aug 17 04:23:07.469880881 (202h56m48.201148501s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470249144 (202h56m48.201516775s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470491772 (202h56m48.201759402s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470580539 (202h56m48.201848169s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470934566 (202h56m48.202202196s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471147235 (202h56m48.202414855s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471257373 (202h56m48.202525003s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471574630 (202h56m48.20284225s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471656044 (202h56m48.202923674s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.472781833 (202h56m48.204049453s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:37.353223355 (202h57m18.084491014s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353345955 (202h57m18.084613605s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353365382 (202h57m18.084633042s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353416519 (202h57m18.084684169s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353477142 (202h57m18.084744793s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 5.161.89.252:26914->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353487252 (202h57m18.084754901s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353530393 (202h57m18.084798093s) [seg_pos={0 0}, upload=211168]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:37.392615539 (202h57m18.123883199s) [upload=211168]: Done uploading segments... Aug 17 04:23:37.392649854 (202h57m18.123917504s) [upload=211168]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 347.564417ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X.XX.....................] duration: 932.141078ms FAIL (2025-08-17 04:39:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 932.165164ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................XXXXXX.....................] duration: 345.723594ms FAIL (2025-08-17 04:39:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:07.365016247 (202h56m48.096283887s) [upload=211168]: Starting upload Aug 17 04:23:07.365113880 (202h56m48.09638152s) [upload=211168]: Done starting upload Aug 17 04:23:07.365185295 (202h56m48.096452915s) [upload=211168]: Uploading segments... Aug 17 04:23:07.365444052 (202h56m48.096711693s) [upload=211168]: Got next segment. Inline: false Aug 17 04:23:07.365496200 (202h56m48.09676384s) [upload=211168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:07.365506590 (202h56m48.096774229s) [upload=211168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:07.391532593 (202h56m48.122800213s) [seg_pos={0 0}, upload=211168]: Begin upload segment... Aug 17 04:23:07.391561247 (202h56m48.122828867s) [seg_pos={0 0}, upload=211168]: Done begin upload segment. Aug 17 04:23:07.391580673 (202h56m48.122848294s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391635377 (202h56m48.122903017s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391895125 (202h56m48.123162745s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392097867 (202h56m48.123365508s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392117784 (202h56m48.123385405s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392313993 (202h56m48.123581614s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392395668 (202h56m48.123663308s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392493893 (202h56m48.123761513s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392681466 (202h56m48.123949106s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392702645 (202h56m48.123970275s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392977794 (202h56m48.124245414s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392988844 (202h56m48.124256475s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393185134 (202h56m48.124452754s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393273310 (202h56m48.12454094s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393400339 (202h56m48.124667959s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393572042 (202h56m48.124839683s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.461635871 (202h56m48.192903501s) [upload=211168]: Next returned nil segment Aug 17 04:23:07.461646972 (202h56m48.192914602s) [upload=211168]: Waiting for error group managing segments... Aug 17 04:23:07.469880881 (202h56m48.201148501s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470249144 (202h56m48.201516775s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470491772 (202h56m48.201759402s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470580539 (202h56m48.201848169s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470934566 (202h56m48.202202196s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471147235 (202h56m48.202414855s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471257373 (202h56m48.202525003s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471574630 (202h56m48.20284225s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471656044 (202h56m48.202923674s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.472781833 (202h56m48.204049453s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:37.353223355 (202h57m18.084491014s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353345955 (202h57m18.084613605s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353365382 (202h57m18.084633042s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353416519 (202h57m18.084684169s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353477142 (202h57m18.084744793s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 5.161.89.252:26914->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353487252 (202h57m18.084754901s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353530393 (202h57m18.084798093s) [seg_pos={0 0}, upload=211168]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:37.392615539 (202h57m18.123883199s) [upload=211168]: Done uploading segments... Aug 17 04:23:37.392649854 (202h57m18.123917504s) [upload=211168]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 345.759441ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.461511877s FAIL (2025-08-17 04:39:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.461537044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XXXX...................] duration: 503.023245ms FAIL (2025-08-17 04:39:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== se Aug 17 04:23:28.620840633 (202h57m9.352108284s) [upload=211175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:28.620845953 (202h57m9.352113603s) [upload=211175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:28.646835158 (202h57m9.378102858s) [seg_pos={0 0}, upload=211175]: Begin upload segment... Aug 17 04:23:28.646936819 (202h57m9.378204468s) [seg_pos={0 0}, upload=211175]: Done begin upload segment. Aug 17 04:23:28.646989559 (202h57m9.378257218s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647051715 (202h57m9.378319375s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647662535 (202h57m9.378930186s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647827836 (202h57m9.379095487s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648134293 (202h57m9.379401954s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648314913 (202h57m9.379582573s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648543414 (202h57m9.379811063s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648834572 (202h57m9.380102222s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648992249 (202h57m9.380259908s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649335063 (202h57m9.380602713s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649507127 (202h57m9.380774777s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649830226 (202h57m9.381097876s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650003341 (202h57m9.381270992s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650256469 (202h57m9.381524128s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650434553 (202h57m9.381702213s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650709621 (202h57m9.381977282s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650857891 (202h57m9.38212554s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.651270246 (202h57m9.382537897s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.768946979 (202h57m9.500214629s) [upload=211175]: Next returned nil segment Aug 17 04:23:28.768974191 (202h57m9.500241841s) [upload=211175]: Waiting for error group managing segments... Aug 17 04:23:28.777572686 (202h57m9.508840347s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.778945461 (202h57m9.510213112s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.785896566 (202h57m9.517164206s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.793497595 (202h57m9.524765245s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.796220951 (202h57m9.527488592s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.799941495 (202h57m9.531209135s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.801171361 (202h57m9.532439001s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.802452863 (202h57m9.533720513s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.822657288 (202h57m9.553924938s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.874167434 (202h57m9.605435084s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.100609905 (202h57m9.831877555s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.216981541 (202h57m9.948249171s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:24:28.606052339 (202h58m9.337319999s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: rpc: dial tcp 104.36.82.44:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606141437 (202h58m9.337409087s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1AowWwVvxzmnW5qwbqbMBVbvNoigffjAN51LQp8e6XA9CZkeXL, address:104.36.82.3:4790): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606261834 (202h58m9.337529484s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:58906->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606338818 (202h58m9.337606469s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:104.36.82.23:4455): protocol: write tcp 5.161.89.252:44290->104.36.82.23:4455: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606369045 (202h58m9.337636696s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: rpc: dial tcp 104.36.82.60:4365: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606443665 (202h58m9.337711316s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, address:104.36.82.49:4003): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606480044 (202h58m9.337747704s) [seg_pos={0 0}, upload=211175]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 04:24:28.621101719 (202h58m9.352369369s) [upload=211175]: Done uploading segments... Aug 17 04:24:28.621117609 (202h58m9.35238526s) [upload=211175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 503.041709ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................XXXX.....................] duration: 331.475324ms FAIL (2025-08-17 04:39:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 04:23:05.076746715 (202h56m45.808014345s) [upload=211167]: Starting upload Aug 17 04:23:05.076819060 (202h56m45.80808669s) [upload=211167]: Done starting upload Aug 17 04:23:05.076858426 (202h56m45.808126045s) [upload=211167]: Uploading segments... Aug 17 04:23:05.077035118 (202h56m45.808302748s) [upload=211167]: Got next segment. Inline: false Aug 17 04:23:05.077065765 (202h56m45.808333395s) [upload=211167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:05.077069302 (202h56m45.808336922s) [upload=211167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:05.149595571 (202h56m45.880863191s) [seg_pos={0 0}, upload=211167]: Begin upload segment... Aug 17 04:23:05.149634344 (202h56m45.880901974s) [seg_pos={0 0}, upload=211167]: Done begin upload segment. Aug 17 04:23:05.149659451 (202h56m45.880927081s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.149699086 (202h56m45.880966716s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150081695 (202h56m45.881349325s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150110339 (202h56m45.88137799s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150390978 (202h56m45.881658608s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150446372 (202h56m45.881714002s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150693598 (202h56m45.881961228s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150755163 (202h56m45.882022793s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150983263 (202h56m45.882250893s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151079885 (202h56m45.882347515s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151274822 (202h56m45.882542452s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151365532 (202h56m45.882633163s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151584305 (202h56m45.882851935s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151603451 (202h56m45.882871081s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.152583125 (202h56m45.883850755s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.153130696 (202h56m45.884398326s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154141749 (202h56m45.88540938s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154561108 (202h56m45.885828738s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.222341082 (202h56m45.953608762s) [upload=211167]: Next returned nil segment Aug 17 04:23:05.222354026 (202h56m45.953621646s) [upload=211167]: Waiting for error group managing segments... Aug 17 04:23:05.230308020 (202h56m45.96157565s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.230754350 (202h56m45.96202198s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231421757 (202h56m45.962689387s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231643444 (202h56m45.962911074s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.296297613 (202h56m46.027565233s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.346576712 (202h56m46.077844332s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.494438107 (202h56m46.225705727s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.563775022 (202h56m46.295042642s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.674589117 (202h56m46.405856747s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.747386525 (202h56m46.478654145s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:24:05.061032967 (202h57m45.792300617s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:104.36.82.60:4351): protocol: rpc: dial tcp 104.36.82.60:4351: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061097619 (202h57m45.79236527s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: rpc: dial tcp 104.36.82.38:4696: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061148564 (202h57m45.792416215s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: rpc: dial tcp 104.36.82.17:4333: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061255446 (202h57m45.792523106s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12Nh9nHAnKWgu1ggCAZ7WcUadbvsPqAgbqzGrBnSaDMqDhquQ4R, address:104.36.82.22:4514): protocol: rpc: dial tcp 104.36.82.22:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061322663 (202h57m45.792590312s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, address:104.36.82.23:4446): protocol: write tcp 5.161.89.252:28480->104.36.82.23:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061363470 (202h57m45.79263112s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: rpc: dial tcp 104.36.82.2:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061387064 (202h57m45.792654714s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u, address:104.36.82.5:4864): protocol: rpc: dial tcp 104.36.82.5:4864: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061424123 (202h57m45.792691784s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA, address:104.36.82.23:4461): protocol: write tcp 5.161.89.252:18768->104.36.82.23:4461: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061453258 (202h57m45.792720908s) [seg_pos={0 0}, upload=211167]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:24:05.085107047 (202h57m45.816374697s) [upload=211167]: Done uploading segments... Aug 17 04:24:05.085125321 (202h57m45.816392971s) [upload=211167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 331.50511ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................XXXXXX....................] duration: 714.758958ms FAIL (2025-08-17 04:38:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:07.365016247 (202h56m48.096283887s) [upload=211168]: Starting upload Aug 17 04:23:07.365113880 (202h56m48.09638152s) [upload=211168]: Done starting upload Aug 17 04:23:07.365185295 (202h56m48.096452915s) [upload=211168]: Uploading segments... Aug 17 04:23:07.365444052 (202h56m48.096711693s) [upload=211168]: Got next segment. Inline: false Aug 17 04:23:07.365496200 (202h56m48.09676384s) [upload=211168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:07.365506590 (202h56m48.096774229s) [upload=211168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:07.391532593 (202h56m48.122800213s) [seg_pos={0 0}, upload=211168]: Begin upload segment... Aug 17 04:23:07.391561247 (202h56m48.122828867s) [seg_pos={0 0}, upload=211168]: Done begin upload segment. Aug 17 04:23:07.391580673 (202h56m48.122848294s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391635377 (202h56m48.122903017s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391895125 (202h56m48.123162745s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392097867 (202h56m48.123365508s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392117784 (202h56m48.123385405s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392313993 (202h56m48.123581614s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392395668 (202h56m48.123663308s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392493893 (202h56m48.123761513s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392681466 (202h56m48.123949106s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392702645 (202h56m48.123970275s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392977794 (202h56m48.124245414s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392988844 (202h56m48.124256475s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393185134 (202h56m48.124452754s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393273310 (202h56m48.12454094s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393400339 (202h56m48.124667959s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393572042 (202h56m48.124839683s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.461635871 (202h56m48.192903501s) [upload=211168]: Next returned nil segment Aug 17 04:23:07.461646972 (202h56m48.192914602s) [upload=211168]: Waiting for error group managing segments... Aug 17 04:23:07.469880881 (202h56m48.201148501s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470249144 (202h56m48.201516775s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470491772 (202h56m48.201759402s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470580539 (202h56m48.201848169s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470934566 (202h56m48.202202196s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471147235 (202h56m48.202414855s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471257373 (202h56m48.202525003s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471574630 (202h56m48.20284225s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471656044 (202h56m48.202923674s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.472781833 (202h56m48.204049453s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:37.353223355 (202h57m18.084491014s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353345955 (202h57m18.084613605s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353365382 (202h57m18.084633042s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353416519 (202h57m18.084684169s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353477142 (202h57m18.084744793s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 5.161.89.252:26914->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353487252 (202h57m18.084754901s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353530393 (202h57m18.084798093s) [seg_pos={0 0}, upload=211168]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:37.392615539 (202h57m18.123883199s) [upload=211168]: Done uploading segments... Aug 17 04:23:37.392649854 (202h57m18.123917504s) [upload=211168]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 714.775109ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................XXXX..................] duration: 447.951773ms FAIL (2025-08-17 04:38:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== se Aug 17 04:23:28.620840633 (202h57m9.352108284s) [upload=211175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:28.620845953 (202h57m9.352113603s) [upload=211175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:28.646835158 (202h57m9.378102858s) [seg_pos={0 0}, upload=211175]: Begin upload segment... Aug 17 04:23:28.646936819 (202h57m9.378204468s) [seg_pos={0 0}, upload=211175]: Done begin upload segment. Aug 17 04:23:28.646989559 (202h57m9.378257218s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647051715 (202h57m9.378319375s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647662535 (202h57m9.378930186s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647827836 (202h57m9.379095487s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648134293 (202h57m9.379401954s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648314913 (202h57m9.379582573s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648543414 (202h57m9.379811063s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648834572 (202h57m9.380102222s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648992249 (202h57m9.380259908s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649335063 (202h57m9.380602713s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649507127 (202h57m9.380774777s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649830226 (202h57m9.381097876s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650003341 (202h57m9.381270992s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650256469 (202h57m9.381524128s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650434553 (202h57m9.381702213s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650709621 (202h57m9.381977282s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650857891 (202h57m9.38212554s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.651270246 (202h57m9.382537897s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.768946979 (202h57m9.500214629s) [upload=211175]: Next returned nil segment Aug 17 04:23:28.768974191 (202h57m9.500241841s) [upload=211175]: Waiting for error group managing segments... Aug 17 04:23:28.777572686 (202h57m9.508840347s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.778945461 (202h57m9.510213112s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.785896566 (202h57m9.517164206s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.793497595 (202h57m9.524765245s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.796220951 (202h57m9.527488592s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.799941495 (202h57m9.531209135s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.801171361 (202h57m9.532439001s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.802452863 (202h57m9.533720513s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.822657288 (202h57m9.553924938s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.874167434 (202h57m9.605435084s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.100609905 (202h57m9.831877555s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.216981541 (202h57m9.948249171s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:24:28.606052339 (202h58m9.337319999s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: rpc: dial tcp 104.36.82.44:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606141437 (202h58m9.337409087s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1AowWwVvxzmnW5qwbqbMBVbvNoigffjAN51LQp8e6XA9CZkeXL, address:104.36.82.3:4790): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606261834 (202h58m9.337529484s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:58906->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606338818 (202h58m9.337606469s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:104.36.82.23:4455): protocol: write tcp 5.161.89.252:44290->104.36.82.23:4455: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606369045 (202h58m9.337636696s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: rpc: dial tcp 104.36.82.60:4365: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606443665 (202h58m9.337711316s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, address:104.36.82.49:4003): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606480044 (202h58m9.337747704s) [seg_pos={0 0}, upload=211175]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 04:24:28.621101719 (202h58m9.352369369s) [upload=211175]: Done uploading segments... Aug 17 04:24:28.621117609 (202h58m9.35238526s) [upload=211175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 447.978084ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X.XX....................] duration: 435.975928ms FAIL (2025-08-17 04:38:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 435.999833ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.475501035s FAIL (2025-08-17 04:38:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.475522307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................XXXXXX...................] duration: 351.255583ms FAIL (2025-08-17 04:38:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:07.365016247 (202h56m48.096283887s) [upload=211168]: Starting upload Aug 17 04:23:07.365113880 (202h56m48.09638152s) [upload=211168]: Done starting upload Aug 17 04:23:07.365185295 (202h56m48.096452915s) [upload=211168]: Uploading segments... Aug 17 04:23:07.365444052 (202h56m48.096711693s) [upload=211168]: Got next segment. Inline: false Aug 17 04:23:07.365496200 (202h56m48.09676384s) [upload=211168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:07.365506590 (202h56m48.096774229s) [upload=211168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:07.391532593 (202h56m48.122800213s) [seg_pos={0 0}, upload=211168]: Begin upload segment... Aug 17 04:23:07.391561247 (202h56m48.122828867s) [seg_pos={0 0}, upload=211168]: Done begin upload segment. Aug 17 04:23:07.391580673 (202h56m48.122848294s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391635377 (202h56m48.122903017s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391895125 (202h56m48.123162745s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392097867 (202h56m48.123365508s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392117784 (202h56m48.123385405s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392313993 (202h56m48.123581614s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392395668 (202h56m48.123663308s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392493893 (202h56m48.123761513s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392681466 (202h56m48.123949106s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392702645 (202h56m48.123970275s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392977794 (202h56m48.124245414s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392988844 (202h56m48.124256475s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393185134 (202h56m48.124452754s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393273310 (202h56m48.12454094s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393400339 (202h56m48.124667959s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393572042 (202h56m48.124839683s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.461635871 (202h56m48.192903501s) [upload=211168]: Next returned nil segment Aug 17 04:23:07.461646972 (202h56m48.192914602s) [upload=211168]: Waiting for error group managing segments... Aug 17 04:23:07.469880881 (202h56m48.201148501s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470249144 (202h56m48.201516775s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470491772 (202h56m48.201759402s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470580539 (202h56m48.201848169s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470934566 (202h56m48.202202196s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471147235 (202h56m48.202414855s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471257373 (202h56m48.202525003s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471574630 (202h56m48.20284225s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471656044 (202h56m48.202923674s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.472781833 (202h56m48.204049453s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:37.353223355 (202h57m18.084491014s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353345955 (202h57m18.084613605s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353365382 (202h57m18.084633042s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353416519 (202h57m18.084684169s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353477142 (202h57m18.084744793s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 5.161.89.252:26914->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353487252 (202h57m18.084754901s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353530393 (202h57m18.084798093s) [seg_pos={0 0}, upload=211168]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:37.392615539 (202h57m18.123883199s) [upload=211168]: Done uploading segments... Aug 17 04:23:37.392649854 (202h57m18.123917504s) [upload=211168]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 351.277094ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XXXX.................] duration: 351.564154ms FAIL (2025-08-17 04:38:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== se Aug 17 04:23:28.620840633 (202h57m9.352108284s) [upload=211175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:28.620845953 (202h57m9.352113603s) [upload=211175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:28.646835158 (202h57m9.378102858s) [seg_pos={0 0}, upload=211175]: Begin upload segment... Aug 17 04:23:28.646936819 (202h57m9.378204468s) [seg_pos={0 0}, upload=211175]: Done begin upload segment. Aug 17 04:23:28.646989559 (202h57m9.378257218s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647051715 (202h57m9.378319375s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647662535 (202h57m9.378930186s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647827836 (202h57m9.379095487s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648134293 (202h57m9.379401954s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648314913 (202h57m9.379582573s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648543414 (202h57m9.379811063s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648834572 (202h57m9.380102222s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648992249 (202h57m9.380259908s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649335063 (202h57m9.380602713s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649507127 (202h57m9.380774777s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649830226 (202h57m9.381097876s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650003341 (202h57m9.381270992s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650256469 (202h57m9.381524128s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650434553 (202h57m9.381702213s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650709621 (202h57m9.381977282s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650857891 (202h57m9.38212554s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.651270246 (202h57m9.382537897s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.768946979 (202h57m9.500214629s) [upload=211175]: Next returned nil segment Aug 17 04:23:28.768974191 (202h57m9.500241841s) [upload=211175]: Waiting for error group managing segments... Aug 17 04:23:28.777572686 (202h57m9.508840347s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.778945461 (202h57m9.510213112s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.785896566 (202h57m9.517164206s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.793497595 (202h57m9.524765245s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.796220951 (202h57m9.527488592s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.799941495 (202h57m9.531209135s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.801171361 (202h57m9.532439001s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.802452863 (202h57m9.533720513s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.822657288 (202h57m9.553924938s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.874167434 (202h57m9.605435084s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.100609905 (202h57m9.831877555s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.216981541 (202h57m9.948249171s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:24:28.606052339 (202h58m9.337319999s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: rpc: dial tcp 104.36.82.44:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606141437 (202h58m9.337409087s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1AowWwVvxzmnW5qwbqbMBVbvNoigffjAN51LQp8e6XA9CZkeXL, address:104.36.82.3:4790): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606261834 (202h58m9.337529484s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:58906->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606338818 (202h58m9.337606469s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:104.36.82.23:4455): protocol: write tcp 5.161.89.252:44290->104.36.82.23:4455: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606369045 (202h58m9.337636696s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: rpc: dial tcp 104.36.82.60:4365: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606443665 (202h58m9.337711316s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, address:104.36.82.49:4003): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606480044 (202h58m9.337747704s) [seg_pos={0 0}, upload=211175]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 04:24:28.621101719 (202h58m9.352369369s) [upload=211175]: Done uploading segments... Aug 17 04:24:28.621117609 (202h58m9.35238526s) [upload=211175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 351.596064ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................XXXX....................] duration: 1.828498766s FAIL (2025-08-17 04:37:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 04:23:05.076746715 (202h56m45.808014345s) [upload=211167]: Starting upload Aug 17 04:23:05.076819060 (202h56m45.80808669s) [upload=211167]: Done starting upload Aug 17 04:23:05.076858426 (202h56m45.808126045s) [upload=211167]: Uploading segments... Aug 17 04:23:05.077035118 (202h56m45.808302748s) [upload=211167]: Got next segment. Inline: false Aug 17 04:23:05.077065765 (202h56m45.808333395s) [upload=211167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:05.077069302 (202h56m45.808336922s) [upload=211167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:05.149595571 (202h56m45.880863191s) [seg_pos={0 0}, upload=211167]: Begin upload segment... Aug 17 04:23:05.149634344 (202h56m45.880901974s) [seg_pos={0 0}, upload=211167]: Done begin upload segment. Aug 17 04:23:05.149659451 (202h56m45.880927081s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.149699086 (202h56m45.880966716s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150081695 (202h56m45.881349325s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150110339 (202h56m45.88137799s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150390978 (202h56m45.881658608s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150446372 (202h56m45.881714002s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150693598 (202h56m45.881961228s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150755163 (202h56m45.882022793s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150983263 (202h56m45.882250893s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151079885 (202h56m45.882347515s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151274822 (202h56m45.882542452s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151365532 (202h56m45.882633163s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151584305 (202h56m45.882851935s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151603451 (202h56m45.882871081s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.152583125 (202h56m45.883850755s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.153130696 (202h56m45.884398326s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154141749 (202h56m45.88540938s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154561108 (202h56m45.885828738s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.222341082 (202h56m45.953608762s) [upload=211167]: Next returned nil segment Aug 17 04:23:05.222354026 (202h56m45.953621646s) [upload=211167]: Waiting for error group managing segments... Aug 17 04:23:05.230308020 (202h56m45.96157565s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.230754350 (202h56m45.96202198s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231421757 (202h56m45.962689387s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231643444 (202h56m45.962911074s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.296297613 (202h56m46.027565233s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.346576712 (202h56m46.077844332s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.494438107 (202h56m46.225705727s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.563775022 (202h56m46.295042642s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.674589117 (202h56m46.405856747s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.747386525 (202h56m46.478654145s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:24:05.061032967 (202h57m45.792300617s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:104.36.82.60:4351): protocol: rpc: dial tcp 104.36.82.60:4351: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061097619 (202h57m45.79236527s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: rpc: dial tcp 104.36.82.38:4696: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061148564 (202h57m45.792416215s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: rpc: dial tcp 104.36.82.17:4333: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061255446 (202h57m45.792523106s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12Nh9nHAnKWgu1ggCAZ7WcUadbvsPqAgbqzGrBnSaDMqDhquQ4R, address:104.36.82.22:4514): protocol: rpc: dial tcp 104.36.82.22:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061322663 (202h57m45.792590312s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, address:104.36.82.23:4446): protocol: write tcp 5.161.89.252:28480->104.36.82.23:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061363470 (202h57m45.79263112s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: rpc: dial tcp 104.36.82.2:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061387064 (202h57m45.792654714s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u, address:104.36.82.5:4864): protocol: rpc: dial tcp 104.36.82.5:4864: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061424123 (202h57m45.792691784s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA, address:104.36.82.23:4461): protocol: write tcp 5.161.89.252:18768->104.36.82.23:4461: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061453258 (202h57m45.792720908s) [seg_pos={0 0}, upload=211167]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:24:05.085107047 (202h57m45.816374697s) [upload=211167]: Done uploading segments... Aug 17 04:24:05.085125321 (202h57m45.816392971s) [upload=211167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1.828553901s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................XXXX................] duration: 494.559379ms FAIL (2025-08-17 04:37:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== se Aug 17 04:23:28.620840633 (202h57m9.352108284s) [upload=211175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:28.620845953 (202h57m9.352113603s) [upload=211175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:28.646835158 (202h57m9.378102858s) [seg_pos={0 0}, upload=211175]: Begin upload segment... Aug 17 04:23:28.646936819 (202h57m9.378204468s) [seg_pos={0 0}, upload=211175]: Done begin upload segment. Aug 17 04:23:28.646989559 (202h57m9.378257218s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647051715 (202h57m9.378319375s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647662535 (202h57m9.378930186s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647827836 (202h57m9.379095487s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648134293 (202h57m9.379401954s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648314913 (202h57m9.379582573s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648543414 (202h57m9.379811063s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648834572 (202h57m9.380102222s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648992249 (202h57m9.380259908s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649335063 (202h57m9.380602713s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649507127 (202h57m9.380774777s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649830226 (202h57m9.381097876s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650003341 (202h57m9.381270992s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650256469 (202h57m9.381524128s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650434553 (202h57m9.381702213s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650709621 (202h57m9.381977282s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650857891 (202h57m9.38212554s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.651270246 (202h57m9.382537897s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.768946979 (202h57m9.500214629s) [upload=211175]: Next returned nil segment Aug 17 04:23:28.768974191 (202h57m9.500241841s) [upload=211175]: Waiting for error group managing segments... Aug 17 04:23:28.777572686 (202h57m9.508840347s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.778945461 (202h57m9.510213112s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.785896566 (202h57m9.517164206s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.793497595 (202h57m9.524765245s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.796220951 (202h57m9.527488592s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.799941495 (202h57m9.531209135s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.801171361 (202h57m9.532439001s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.802452863 (202h57m9.533720513s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.822657288 (202h57m9.553924938s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.874167434 (202h57m9.605435084s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.100609905 (202h57m9.831877555s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.216981541 (202h57m9.948249171s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:24:28.606052339 (202h58m9.337319999s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: rpc: dial tcp 104.36.82.44:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606141437 (202h58m9.337409087s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1AowWwVvxzmnW5qwbqbMBVbvNoigffjAN51LQp8e6XA9CZkeXL, address:104.36.82.3:4790): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606261834 (202h58m9.337529484s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:58906->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606338818 (202h58m9.337606469s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:104.36.82.23:4455): protocol: write tcp 5.161.89.252:44290->104.36.82.23:4455: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606369045 (202h58m9.337636696s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: rpc: dial tcp 104.36.82.60:4365: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606443665 (202h58m9.337711316s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, address:104.36.82.49:4003): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606480044 (202h58m9.337747704s) [seg_pos={0 0}, upload=211175]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 04:24:28.621101719 (202h58m9.352369369s) [upload=211175]: Done uploading segments... Aug 17 04:24:28.621117609 (202h58m9.35238526s) [upload=211175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 494.582292ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XXXX...................] duration: 382.351622ms FAIL (2025-08-17 04:37:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 04:23:05.076746715 (202h56m45.808014345s) [upload=211167]: Starting upload Aug 17 04:23:05.076819060 (202h56m45.80808669s) [upload=211167]: Done starting upload Aug 17 04:23:05.076858426 (202h56m45.808126045s) [upload=211167]: Uploading segments... Aug 17 04:23:05.077035118 (202h56m45.808302748s) [upload=211167]: Got next segment. Inline: false Aug 17 04:23:05.077065765 (202h56m45.808333395s) [upload=211167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:05.077069302 (202h56m45.808336922s) [upload=211167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:05.149595571 (202h56m45.880863191s) [seg_pos={0 0}, upload=211167]: Begin upload segment... Aug 17 04:23:05.149634344 (202h56m45.880901974s) [seg_pos={0 0}, upload=211167]: Done begin upload segment. Aug 17 04:23:05.149659451 (202h56m45.880927081s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.149699086 (202h56m45.880966716s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150081695 (202h56m45.881349325s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150110339 (202h56m45.88137799s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150390978 (202h56m45.881658608s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150446372 (202h56m45.881714002s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150693598 (202h56m45.881961228s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150755163 (202h56m45.882022793s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150983263 (202h56m45.882250893s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151079885 (202h56m45.882347515s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151274822 (202h56m45.882542452s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151365532 (202h56m45.882633163s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151584305 (202h56m45.882851935s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151603451 (202h56m45.882871081s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.152583125 (202h56m45.883850755s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.153130696 (202h56m45.884398326s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154141749 (202h56m45.88540938s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154561108 (202h56m45.885828738s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.222341082 (202h56m45.953608762s) [upload=211167]: Next returned nil segment Aug 17 04:23:05.222354026 (202h56m45.953621646s) [upload=211167]: Waiting for error group managing segments... Aug 17 04:23:05.230308020 (202h56m45.96157565s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.230754350 (202h56m45.96202198s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231421757 (202h56m45.962689387s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231643444 (202h56m45.962911074s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.296297613 (202h56m46.027565233s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.346576712 (202h56m46.077844332s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.494438107 (202h56m46.225705727s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.563775022 (202h56m46.295042642s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.674589117 (202h56m46.405856747s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.747386525 (202h56m46.478654145s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:24:05.061032967 (202h57m45.792300617s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:104.36.82.60:4351): protocol: rpc: dial tcp 104.36.82.60:4351: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061097619 (202h57m45.79236527s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: rpc: dial tcp 104.36.82.38:4696: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061148564 (202h57m45.792416215s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: rpc: dial tcp 104.36.82.17:4333: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061255446 (202h57m45.792523106s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12Nh9nHAnKWgu1ggCAZ7WcUadbvsPqAgbqzGrBnSaDMqDhquQ4R, address:104.36.82.22:4514): protocol: rpc: dial tcp 104.36.82.22:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061322663 (202h57m45.792590312s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, address:104.36.82.23:4446): protocol: write tcp 5.161.89.252:28480->104.36.82.23:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061363470 (202h57m45.79263112s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: rpc: dial tcp 104.36.82.2:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061387064 (202h57m45.792654714s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u, address:104.36.82.5:4864): protocol: rpc: dial tcp 104.36.82.5:4864: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061424123 (202h57m45.792691784s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA, address:104.36.82.23:4461): protocol: write tcp 5.161.89.252:18768->104.36.82.23:4461: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061453258 (202h57m45.792720908s) [seg_pos={0 0}, upload=211167]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:24:05.085107047 (202h57m45.816374697s) [upload=211167]: Done uploading segments... Aug 17 04:24:05.085125321 (202h57m45.816392971s) [upload=211167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 382.374916ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.389666375s FAIL (2025-08-17 04:37:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.389685613s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X.XX...................] duration: 729.827018ms FAIL (2025-08-17 04:37:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 729.842227ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................................XXXXXX..................] duration: 319.863957ms FAIL (2025-08-17 04:37:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:07.365016247 (202h56m48.096283887s) [upload=211168]: Starting upload Aug 17 04:23:07.365113880 (202h56m48.09638152s) [upload=211168]: Done starting upload Aug 17 04:23:07.365185295 (202h56m48.096452915s) [upload=211168]: Uploading segments... Aug 17 04:23:07.365444052 (202h56m48.096711693s) [upload=211168]: Got next segment. Inline: false Aug 17 04:23:07.365496200 (202h56m48.09676384s) [upload=211168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:07.365506590 (202h56m48.096774229s) [upload=211168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:07.391532593 (202h56m48.122800213s) [seg_pos={0 0}, upload=211168]: Begin upload segment... Aug 17 04:23:07.391561247 (202h56m48.122828867s) [seg_pos={0 0}, upload=211168]: Done begin upload segment. Aug 17 04:23:07.391580673 (202h56m48.122848294s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391635377 (202h56m48.122903017s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391895125 (202h56m48.123162745s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392097867 (202h56m48.123365508s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392117784 (202h56m48.123385405s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392313993 (202h56m48.123581614s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392395668 (202h56m48.123663308s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392493893 (202h56m48.123761513s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392681466 (202h56m48.123949106s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392702645 (202h56m48.123970275s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392977794 (202h56m48.124245414s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392988844 (202h56m48.124256475s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393185134 (202h56m48.124452754s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393273310 (202h56m48.12454094s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393400339 (202h56m48.124667959s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393572042 (202h56m48.124839683s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.461635871 (202h56m48.192903501s) [upload=211168]: Next returned nil segment Aug 17 04:23:07.461646972 (202h56m48.192914602s) [upload=211168]: Waiting for error group managing segments... Aug 17 04:23:07.469880881 (202h56m48.201148501s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470249144 (202h56m48.201516775s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470491772 (202h56m48.201759402s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470580539 (202h56m48.201848169s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470934566 (202h56m48.202202196s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471147235 (202h56m48.202414855s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471257373 (202h56m48.202525003s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471574630 (202h56m48.20284225s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471656044 (202h56m48.202923674s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.472781833 (202h56m48.204049453s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:37.353223355 (202h57m18.084491014s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353345955 (202h57m18.084613605s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353365382 (202h57m18.084633042s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353416519 (202h57m18.084684169s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353477142 (202h57m18.084744793s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 5.161.89.252:26914->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353487252 (202h57m18.084754901s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353530393 (202h57m18.084798093s) [seg_pos={0 0}, upload=211168]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:37.392615539 (202h57m18.123883199s) [upload=211168]: Done uploading segments... Aug 17 04:23:37.392649854 (202h57m18.123917504s) [upload=211168]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 319.880707ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.532654174s FAIL (2025-08-17 04:36:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.532673381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................XXXXXX.................] duration: 1.165617103s FAIL (2025-08-17 04:36:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:07.365016247 (202h56m48.096283887s) [upload=211168]: Starting upload Aug 17 04:23:07.365113880 (202h56m48.09638152s) [upload=211168]: Done starting upload Aug 17 04:23:07.365185295 (202h56m48.096452915s) [upload=211168]: Uploading segments... Aug 17 04:23:07.365444052 (202h56m48.096711693s) [upload=211168]: Got next segment. Inline: false Aug 17 04:23:07.365496200 (202h56m48.09676384s) [upload=211168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:07.365506590 (202h56m48.096774229s) [upload=211168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:07.391532593 (202h56m48.122800213s) [seg_pos={0 0}, upload=211168]: Begin upload segment... Aug 17 04:23:07.391561247 (202h56m48.122828867s) [seg_pos={0 0}, upload=211168]: Done begin upload segment. Aug 17 04:23:07.391580673 (202h56m48.122848294s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391635377 (202h56m48.122903017s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391895125 (202h56m48.123162745s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392097867 (202h56m48.123365508s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392117784 (202h56m48.123385405s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392313993 (202h56m48.123581614s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392395668 (202h56m48.123663308s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392493893 (202h56m48.123761513s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392681466 (202h56m48.123949106s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392702645 (202h56m48.123970275s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392977794 (202h56m48.124245414s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392988844 (202h56m48.124256475s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393185134 (202h56m48.124452754s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393273310 (202h56m48.12454094s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393400339 (202h56m48.124667959s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393572042 (202h56m48.124839683s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.461635871 (202h56m48.192903501s) [upload=211168]: Next returned nil segment Aug 17 04:23:07.461646972 (202h56m48.192914602s) [upload=211168]: Waiting for error group managing segments... Aug 17 04:23:07.469880881 (202h56m48.201148501s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470249144 (202h56m48.201516775s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470491772 (202h56m48.201759402s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470580539 (202h56m48.201848169s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470934566 (202h56m48.202202196s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471147235 (202h56m48.202414855s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471257373 (202h56m48.202525003s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471574630 (202h56m48.20284225s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471656044 (202h56m48.202923674s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.472781833 (202h56m48.204049453s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:37.353223355 (202h57m18.084491014s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353345955 (202h57m18.084613605s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353365382 (202h57m18.084633042s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353416519 (202h57m18.084684169s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353477142 (202h57m18.084744793s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 5.161.89.252:26914->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353487252 (202h57m18.084754901s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353530393 (202h57m18.084798093s) [seg_pos={0 0}, upload=211168]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:37.392615539 (202h57m18.123883199s) [upload=211168]: Done uploading segments... Aug 17 04:23:37.392649854 (202h57m18.123917504s) [upload=211168]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.165644586s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................XXXX...............] duration: 438.530115ms FAIL (2025-08-17 04:36:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== se Aug 17 04:23:28.620840633 (202h57m9.352108284s) [upload=211175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:28.620845953 (202h57m9.352113603s) [upload=211175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:28.646835158 (202h57m9.378102858s) [seg_pos={0 0}, upload=211175]: Begin upload segment... Aug 17 04:23:28.646936819 (202h57m9.378204468s) [seg_pos={0 0}, upload=211175]: Done begin upload segment. Aug 17 04:23:28.646989559 (202h57m9.378257218s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647051715 (202h57m9.378319375s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647662535 (202h57m9.378930186s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647827836 (202h57m9.379095487s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648134293 (202h57m9.379401954s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648314913 (202h57m9.379582573s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648543414 (202h57m9.379811063s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648834572 (202h57m9.380102222s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648992249 (202h57m9.380259908s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649335063 (202h57m9.380602713s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649507127 (202h57m9.380774777s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649830226 (202h57m9.381097876s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650003341 (202h57m9.381270992s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650256469 (202h57m9.381524128s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650434553 (202h57m9.381702213s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650709621 (202h57m9.381977282s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650857891 (202h57m9.38212554s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.651270246 (202h57m9.382537897s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.768946979 (202h57m9.500214629s) [upload=211175]: Next returned nil segment Aug 17 04:23:28.768974191 (202h57m9.500241841s) [upload=211175]: Waiting for error group managing segments... Aug 17 04:23:28.777572686 (202h57m9.508840347s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.778945461 (202h57m9.510213112s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.785896566 (202h57m9.517164206s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.793497595 (202h57m9.524765245s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.796220951 (202h57m9.527488592s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.799941495 (202h57m9.531209135s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.801171361 (202h57m9.532439001s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.802452863 (202h57m9.533720513s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.822657288 (202h57m9.553924938s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.874167434 (202h57m9.605435084s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.100609905 (202h57m9.831877555s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.216981541 (202h57m9.948249171s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:24:28.606052339 (202h58m9.337319999s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: rpc: dial tcp 104.36.82.44:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606141437 (202h58m9.337409087s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1AowWwVvxzmnW5qwbqbMBVbvNoigffjAN51LQp8e6XA9CZkeXL, address:104.36.82.3:4790): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606261834 (202h58m9.337529484s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:58906->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606338818 (202h58m9.337606469s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:104.36.82.23:4455): protocol: write tcp 5.161.89.252:44290->104.36.82.23:4455: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606369045 (202h58m9.337636696s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: rpc: dial tcp 104.36.82.60:4365: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606443665 (202h58m9.337711316s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, address:104.36.82.49:4003): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606480044 (202h58m9.337747704s) [seg_pos={0 0}, upload=211175]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 04:24:28.621101719 (202h58m9.352369369s) [upload=211175]: Done uploading segments... Aug 17 04:24:28.621117609 (202h58m9.35238526s) [upload=211175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 438.560872ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................XXXX..................] duration: 477.301505ms FAIL (2025-08-17 04:36:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 04:23:05.076746715 (202h56m45.808014345s) [upload=211167]: Starting upload Aug 17 04:23:05.076819060 (202h56m45.80808669s) [upload=211167]: Done starting upload Aug 17 04:23:05.076858426 (202h56m45.808126045s) [upload=211167]: Uploading segments... Aug 17 04:23:05.077035118 (202h56m45.808302748s) [upload=211167]: Got next segment. Inline: false Aug 17 04:23:05.077065765 (202h56m45.808333395s) [upload=211167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:05.077069302 (202h56m45.808336922s) [upload=211167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:05.149595571 (202h56m45.880863191s) [seg_pos={0 0}, upload=211167]: Begin upload segment... Aug 17 04:23:05.149634344 (202h56m45.880901974s) [seg_pos={0 0}, upload=211167]: Done begin upload segment. Aug 17 04:23:05.149659451 (202h56m45.880927081s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.149699086 (202h56m45.880966716s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150081695 (202h56m45.881349325s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150110339 (202h56m45.88137799s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150390978 (202h56m45.881658608s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150446372 (202h56m45.881714002s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150693598 (202h56m45.881961228s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150755163 (202h56m45.882022793s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150983263 (202h56m45.882250893s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151079885 (202h56m45.882347515s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151274822 (202h56m45.882542452s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151365532 (202h56m45.882633163s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151584305 (202h56m45.882851935s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151603451 (202h56m45.882871081s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.152583125 (202h56m45.883850755s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.153130696 (202h56m45.884398326s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154141749 (202h56m45.88540938s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154561108 (202h56m45.885828738s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.222341082 (202h56m45.953608762s) [upload=211167]: Next returned nil segment Aug 17 04:23:05.222354026 (202h56m45.953621646s) [upload=211167]: Waiting for error group managing segments... Aug 17 04:23:05.230308020 (202h56m45.96157565s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.230754350 (202h56m45.96202198s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231421757 (202h56m45.962689387s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231643444 (202h56m45.962911074s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.296297613 (202h56m46.027565233s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.346576712 (202h56m46.077844332s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.494438107 (202h56m46.225705727s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.563775022 (202h56m46.295042642s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.674589117 (202h56m46.405856747s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.747386525 (202h56m46.478654145s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:24:05.061032967 (202h57m45.792300617s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:104.36.82.60:4351): protocol: rpc: dial tcp 104.36.82.60:4351: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061097619 (202h57m45.79236527s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: rpc: dial tcp 104.36.82.38:4696: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061148564 (202h57m45.792416215s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: rpc: dial tcp 104.36.82.17:4333: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061255446 (202h57m45.792523106s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12Nh9nHAnKWgu1ggCAZ7WcUadbvsPqAgbqzGrBnSaDMqDhquQ4R, address:104.36.82.22:4514): protocol: rpc: dial tcp 104.36.82.22:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061322663 (202h57m45.792590312s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, address:104.36.82.23:4446): protocol: write tcp 5.161.89.252:28480->104.36.82.23:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061363470 (202h57m45.79263112s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: rpc: dial tcp 104.36.82.2:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061387064 (202h57m45.792654714s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u, address:104.36.82.5:4864): protocol: rpc: dial tcp 104.36.82.5:4864: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061424123 (202h57m45.792691784s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA, address:104.36.82.23:4461): protocol: write tcp 5.161.89.252:18768->104.36.82.23:4461: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061453258 (202h57m45.792720908s) [seg_pos={0 0}, upload=211167]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:24:05.085107047 (202h57m45.816374697s) [upload=211167]: Done uploading segments... Aug 17 04:24:05.085125321 (202h57m45.816392971s) [upload=211167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 477.335269ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X.XX..................] duration: 384.351444ms FAIL (2025-08-17 04:36:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 384.372944ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................XXXXXX................] duration: 385.856155ms FAIL (2025-08-17 04:36:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:07.365016247 (202h56m48.096283887s) [upload=211168]: Starting upload Aug 17 04:23:07.365113880 (202h56m48.09638152s) [upload=211168]: Done starting upload Aug 17 04:23:07.365185295 (202h56m48.096452915s) [upload=211168]: Uploading segments... Aug 17 04:23:07.365444052 (202h56m48.096711693s) [upload=211168]: Got next segment. Inline: false Aug 17 04:23:07.365496200 (202h56m48.09676384s) [upload=211168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:07.365506590 (202h56m48.096774229s) [upload=211168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:07.391532593 (202h56m48.122800213s) [seg_pos={0 0}, upload=211168]: Begin upload segment... Aug 17 04:23:07.391561247 (202h56m48.122828867s) [seg_pos={0 0}, upload=211168]: Done begin upload segment. Aug 17 04:23:07.391580673 (202h56m48.122848294s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391635377 (202h56m48.122903017s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391895125 (202h56m48.123162745s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392097867 (202h56m48.123365508s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392117784 (202h56m48.123385405s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392313993 (202h56m48.123581614s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392395668 (202h56m48.123663308s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392493893 (202h56m48.123761513s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392681466 (202h56m48.123949106s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392702645 (202h56m48.123970275s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392977794 (202h56m48.124245414s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392988844 (202h56m48.124256475s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393185134 (202h56m48.124452754s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393273310 (202h56m48.12454094s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393400339 (202h56m48.124667959s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393572042 (202h56m48.124839683s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.461635871 (202h56m48.192903501s) [upload=211168]: Next returned nil segment Aug 17 04:23:07.461646972 (202h56m48.192914602s) [upload=211168]: Waiting for error group managing segments... Aug 17 04:23:07.469880881 (202h56m48.201148501s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470249144 (202h56m48.201516775s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470491772 (202h56m48.201759402s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470580539 (202h56m48.201848169s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470934566 (202h56m48.202202196s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471147235 (202h56m48.202414855s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471257373 (202h56m48.202525003s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471574630 (202h56m48.20284225s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471656044 (202h56m48.202923674s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.472781833 (202h56m48.204049453s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:37.353223355 (202h57m18.084491014s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353345955 (202h57m18.084613605s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353365382 (202h57m18.084633042s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353416519 (202h57m18.084684169s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353477142 (202h57m18.084744793s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 5.161.89.252:26914->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353487252 (202h57m18.084754901s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353530393 (202h57m18.084798093s) [seg_pos={0 0}, upload=211168]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:37.392615539 (202h57m18.123883199s) [upload=211168]: Done uploading segments... Aug 17 04:23:37.392649854 (202h57m18.123917504s) [upload=211168]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 385.888776ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X.XX.................] duration: 405.741771ms FAIL (2025-08-17 04:35:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 405.757671ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................XXXX..............] duration: 400.704292ms FAIL (2025-08-17 04:35:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== se Aug 17 04:23:28.620840633 (202h57m9.352108284s) [upload=211175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:28.620845953 (202h57m9.352113603s) [upload=211175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:28.646835158 (202h57m9.378102858s) [seg_pos={0 0}, upload=211175]: Begin upload segment... Aug 17 04:23:28.646936819 (202h57m9.378204468s) [seg_pos={0 0}, upload=211175]: Done begin upload segment. Aug 17 04:23:28.646989559 (202h57m9.378257218s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647051715 (202h57m9.378319375s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647662535 (202h57m9.378930186s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647827836 (202h57m9.379095487s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648134293 (202h57m9.379401954s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648314913 (202h57m9.379582573s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648543414 (202h57m9.379811063s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648834572 (202h57m9.380102222s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648992249 (202h57m9.380259908s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649335063 (202h57m9.380602713s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649507127 (202h57m9.380774777s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649830226 (202h57m9.381097876s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650003341 (202h57m9.381270992s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650256469 (202h57m9.381524128s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650434553 (202h57m9.381702213s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650709621 (202h57m9.381977282s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650857891 (202h57m9.38212554s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.651270246 (202h57m9.382537897s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.768946979 (202h57m9.500214629s) [upload=211175]: Next returned nil segment Aug 17 04:23:28.768974191 (202h57m9.500241841s) [upload=211175]: Waiting for error group managing segments... Aug 17 04:23:28.777572686 (202h57m9.508840347s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.778945461 (202h57m9.510213112s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.785896566 (202h57m9.517164206s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.793497595 (202h57m9.524765245s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.796220951 (202h57m9.527488592s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.799941495 (202h57m9.531209135s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.801171361 (202h57m9.532439001s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.802452863 (202h57m9.533720513s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.822657288 (202h57m9.553924938s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.874167434 (202h57m9.605435084s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.100609905 (202h57m9.831877555s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.216981541 (202h57m9.948249171s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:24:28.606052339 (202h58m9.337319999s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: rpc: dial tcp 104.36.82.44:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606141437 (202h58m9.337409087s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1AowWwVvxzmnW5qwbqbMBVbvNoigffjAN51LQp8e6XA9CZkeXL, address:104.36.82.3:4790): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606261834 (202h58m9.337529484s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:58906->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606338818 (202h58m9.337606469s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:104.36.82.23:4455): protocol: write tcp 5.161.89.252:44290->104.36.82.23:4455: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606369045 (202h58m9.337636696s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: rpc: dial tcp 104.36.82.60:4365: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606443665 (202h58m9.337711316s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, address:104.36.82.49:4003): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606480044 (202h58m9.337747704s) [seg_pos={0 0}, upload=211175]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 04:24:28.621101719 (202h58m9.352369369s) [upload=211175]: Done uploading segments... Aug 17 04:24:28.621117609 (202h58m9.35238526s) [upload=211175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 400.728438ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.30179648s FAIL (2025-08-17 04:35:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.301816377s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XXXX.................] duration: 395.933246ms FAIL (2025-08-17 04:35:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 04:23:05.076746715 (202h56m45.808014345s) [upload=211167]: Starting upload Aug 17 04:23:05.076819060 (202h56m45.80808669s) [upload=211167]: Done starting upload Aug 17 04:23:05.076858426 (202h56m45.808126045s) [upload=211167]: Uploading segments... Aug 17 04:23:05.077035118 (202h56m45.808302748s) [upload=211167]: Got next segment. Inline: false Aug 17 04:23:05.077065765 (202h56m45.808333395s) [upload=211167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:05.077069302 (202h56m45.808336922s) [upload=211167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:05.149595571 (202h56m45.880863191s) [seg_pos={0 0}, upload=211167]: Begin upload segment... Aug 17 04:23:05.149634344 (202h56m45.880901974s) [seg_pos={0 0}, upload=211167]: Done begin upload segment. Aug 17 04:23:05.149659451 (202h56m45.880927081s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.149699086 (202h56m45.880966716s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150081695 (202h56m45.881349325s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150110339 (202h56m45.88137799s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150390978 (202h56m45.881658608s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150446372 (202h56m45.881714002s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150693598 (202h56m45.881961228s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150755163 (202h56m45.882022793s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150983263 (202h56m45.882250893s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151079885 (202h56m45.882347515s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151274822 (202h56m45.882542452s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151365532 (202h56m45.882633163s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151584305 (202h56m45.882851935s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151603451 (202h56m45.882871081s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.152583125 (202h56m45.883850755s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.153130696 (202h56m45.884398326s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154141749 (202h56m45.88540938s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154561108 (202h56m45.885828738s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.222341082 (202h56m45.953608762s) [upload=211167]: Next returned nil segment Aug 17 04:23:05.222354026 (202h56m45.953621646s) [upload=211167]: Waiting for error group managing segments... Aug 17 04:23:05.230308020 (202h56m45.96157565s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.230754350 (202h56m45.96202198s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231421757 (202h56m45.962689387s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231643444 (202h56m45.962911074s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.296297613 (202h56m46.027565233s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.346576712 (202h56m46.077844332s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.494438107 (202h56m46.225705727s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.563775022 (202h56m46.295042642s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.674589117 (202h56m46.405856747s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.747386525 (202h56m46.478654145s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:24:05.061032967 (202h57m45.792300617s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:104.36.82.60:4351): protocol: rpc: dial tcp 104.36.82.60:4351: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061097619 (202h57m45.79236527s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: rpc: dial tcp 104.36.82.38:4696: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061148564 (202h57m45.792416215s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: rpc: dial tcp 104.36.82.17:4333: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061255446 (202h57m45.792523106s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12Nh9nHAnKWgu1ggCAZ7WcUadbvsPqAgbqzGrBnSaDMqDhquQ4R, address:104.36.82.22:4514): protocol: rpc: dial tcp 104.36.82.22:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061322663 (202h57m45.792590312s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, address:104.36.82.23:4446): protocol: write tcp 5.161.89.252:28480->104.36.82.23:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061363470 (202h57m45.79263112s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: rpc: dial tcp 104.36.82.2:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061387064 (202h57m45.792654714s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u, address:104.36.82.5:4864): protocol: rpc: dial tcp 104.36.82.5:4864: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061424123 (202h57m45.792691784s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA, address:104.36.82.23:4461): protocol: write tcp 5.161.89.252:18768->104.36.82.23:4461: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061453258 (202h57m45.792720908s) [seg_pos={0 0}, upload=211167]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:24:05.085107047 (202h57m45.816374697s) [upload=211167]: Done uploading segments... Aug 17 04:24:05.085125321 (202h57m45.816392971s) [upload=211167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 395.959486ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................................XXXXXX...............] duration: 434.267304ms FAIL (2025-08-17 04:35:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:07.365016247 (202h56m48.096283887s) [upload=211168]: Starting upload Aug 17 04:23:07.365113880 (202h56m48.09638152s) [upload=211168]: Done starting upload Aug 17 04:23:07.365185295 (202h56m48.096452915s) [upload=211168]: Uploading segments... Aug 17 04:23:07.365444052 (202h56m48.096711693s) [upload=211168]: Got next segment. Inline: false Aug 17 04:23:07.365496200 (202h56m48.09676384s) [upload=211168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:07.365506590 (202h56m48.096774229s) [upload=211168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:07.391532593 (202h56m48.122800213s) [seg_pos={0 0}, upload=211168]: Begin upload segment... Aug 17 04:23:07.391561247 (202h56m48.122828867s) [seg_pos={0 0}, upload=211168]: Done begin upload segment. Aug 17 04:23:07.391580673 (202h56m48.122848294s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391635377 (202h56m48.122903017s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391895125 (202h56m48.123162745s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392097867 (202h56m48.123365508s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392117784 (202h56m48.123385405s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392313993 (202h56m48.123581614s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392395668 (202h56m48.123663308s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392493893 (202h56m48.123761513s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392681466 (202h56m48.123949106s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392702645 (202h56m48.123970275s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392977794 (202h56m48.124245414s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392988844 (202h56m48.124256475s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393185134 (202h56m48.124452754s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393273310 (202h56m48.12454094s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393400339 (202h56m48.124667959s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393572042 (202h56m48.124839683s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.461635871 (202h56m48.192903501s) [upload=211168]: Next returned nil segment Aug 17 04:23:07.461646972 (202h56m48.192914602s) [upload=211168]: Waiting for error group managing segments... Aug 17 04:23:07.469880881 (202h56m48.201148501s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470249144 (202h56m48.201516775s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470491772 (202h56m48.201759402s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470580539 (202h56m48.201848169s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470934566 (202h56m48.202202196s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471147235 (202h56m48.202414855s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471257373 (202h56m48.202525003s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471574630 (202h56m48.20284225s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471656044 (202h56m48.202923674s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.472781833 (202h56m48.204049453s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:37.353223355 (202h57m18.084491014s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353345955 (202h57m18.084613605s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353365382 (202h57m18.084633042s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353416519 (202h57m18.084684169s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353477142 (202h57m18.084744793s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 5.161.89.252:26914->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353487252 (202h57m18.084754901s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353530393 (202h57m18.084798093s) [seg_pos={0 0}, upload=211168]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:37.392615539 (202h57m18.123883199s) [upload=211168]: Done uploading segments... Aug 17 04:23:37.392649854 (202h57m18.123917504s) [upload=211168]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 434.292072ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X.XX................] duration: 358.859059ms FAIL (2025-08-17 04:34:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 358.872263ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.785566776s FAIL (2025-08-17 04:34:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.785586455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................................XXXXXX..............] duration: 317.117368ms FAIL (2025-08-17 04:34:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:07.365016247 (202h56m48.096283887s) [upload=211168]: Starting upload Aug 17 04:23:07.365113880 (202h56m48.09638152s) [upload=211168]: Done starting upload Aug 17 04:23:07.365185295 (202h56m48.096452915s) [upload=211168]: Uploading segments... Aug 17 04:23:07.365444052 (202h56m48.096711693s) [upload=211168]: Got next segment. Inline: false Aug 17 04:23:07.365496200 (202h56m48.09676384s) [upload=211168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:07.365506590 (202h56m48.096774229s) [upload=211168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:07.391532593 (202h56m48.122800213s) [seg_pos={0 0}, upload=211168]: Begin upload segment... Aug 17 04:23:07.391561247 (202h56m48.122828867s) [seg_pos={0 0}, upload=211168]: Done begin upload segment. Aug 17 04:23:07.391580673 (202h56m48.122848294s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391635377 (202h56m48.122903017s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391895125 (202h56m48.123162745s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392097867 (202h56m48.123365508s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392117784 (202h56m48.123385405s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392313993 (202h56m48.123581614s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392395668 (202h56m48.123663308s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392493893 (202h56m48.123761513s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392681466 (202h56m48.123949106s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392702645 (202h56m48.123970275s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392977794 (202h56m48.124245414s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392988844 (202h56m48.124256475s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393185134 (202h56m48.124452754s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393273310 (202h56m48.12454094s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393400339 (202h56m48.124667959s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393572042 (202h56m48.124839683s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.461635871 (202h56m48.192903501s) [upload=211168]: Next returned nil segment Aug 17 04:23:07.461646972 (202h56m48.192914602s) [upload=211168]: Waiting for error group managing segments... Aug 17 04:23:07.469880881 (202h56m48.201148501s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470249144 (202h56m48.201516775s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470491772 (202h56m48.201759402s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470580539 (202h56m48.201848169s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470934566 (202h56m48.202202196s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471147235 (202h56m48.202414855s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471257373 (202h56m48.202525003s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471574630 (202h56m48.20284225s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471656044 (202h56m48.202923674s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.472781833 (202h56m48.204049453s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:37.353223355 (202h57m18.084491014s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353345955 (202h57m18.084613605s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353365382 (202h57m18.084633042s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353416519 (202h57m18.084684169s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353477142 (202h57m18.084744793s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 5.161.89.252:26914->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353487252 (202h57m18.084754901s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353530393 (202h57m18.084798093s) [seg_pos={0 0}, upload=211168]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:37.392615539 (202h57m18.123883199s) [upload=211168]: Done uploading segments... Aug 17 04:23:37.392649854 (202h57m18.123917504s) [upload=211168]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 317.142476ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................XXXX.............] duration: 586.646476ms FAIL (2025-08-17 04:34:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== se Aug 17 04:23:28.620840633 (202h57m9.352108284s) [upload=211175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:28.620845953 (202h57m9.352113603s) [upload=211175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:28.646835158 (202h57m9.378102858s) [seg_pos={0 0}, upload=211175]: Begin upload segment... Aug 17 04:23:28.646936819 (202h57m9.378204468s) [seg_pos={0 0}, upload=211175]: Done begin upload segment. Aug 17 04:23:28.646989559 (202h57m9.378257218s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647051715 (202h57m9.378319375s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647662535 (202h57m9.378930186s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647827836 (202h57m9.379095487s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648134293 (202h57m9.379401954s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648314913 (202h57m9.379582573s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648543414 (202h57m9.379811063s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648834572 (202h57m9.380102222s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648992249 (202h57m9.380259908s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649335063 (202h57m9.380602713s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649507127 (202h57m9.380774777s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649830226 (202h57m9.381097876s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650003341 (202h57m9.381270992s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650256469 (202h57m9.381524128s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650434553 (202h57m9.381702213s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650709621 (202h57m9.381977282s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650857891 (202h57m9.38212554s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.651270246 (202h57m9.382537897s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.768946979 (202h57m9.500214629s) [upload=211175]: Next returned nil segment Aug 17 04:23:28.768974191 (202h57m9.500241841s) [upload=211175]: Waiting for error group managing segments... Aug 17 04:23:28.777572686 (202h57m9.508840347s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.778945461 (202h57m9.510213112s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.785896566 (202h57m9.517164206s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.793497595 (202h57m9.524765245s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.796220951 (202h57m9.527488592s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.799941495 (202h57m9.531209135s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.801171361 (202h57m9.532439001s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.802452863 (202h57m9.533720513s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.822657288 (202h57m9.553924938s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.874167434 (202h57m9.605435084s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.100609905 (202h57m9.831877555s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.216981541 (202h57m9.948249171s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:24:28.606052339 (202h58m9.337319999s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: rpc: dial tcp 104.36.82.44:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606141437 (202h58m9.337409087s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1AowWwVvxzmnW5qwbqbMBVbvNoigffjAN51LQp8e6XA9CZkeXL, address:104.36.82.3:4790): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606261834 (202h58m9.337529484s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:58906->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606338818 (202h58m9.337606469s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:104.36.82.23:4455): protocol: write tcp 5.161.89.252:44290->104.36.82.23:4455: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606369045 (202h58m9.337636696s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: rpc: dial tcp 104.36.82.60:4365: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606443665 (202h58m9.337711316s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, address:104.36.82.49:4003): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606480044 (202h58m9.337747704s) [seg_pos={0 0}, upload=211175]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 04:24:28.621101719 (202h58m9.352369369s) [upload=211175]: Done uploading segments... Aug 17 04:24:28.621117609 (202h58m9.35238526s) [upload=211175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 586.670141ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................XXXX................] duration: 446.979683ms FAIL (2025-08-17 04:34:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 04:23:05.076746715 (202h56m45.808014345s) [upload=211167]: Starting upload Aug 17 04:23:05.076819060 (202h56m45.80808669s) [upload=211167]: Done starting upload Aug 17 04:23:05.076858426 (202h56m45.808126045s) [upload=211167]: Uploading segments... Aug 17 04:23:05.077035118 (202h56m45.808302748s) [upload=211167]: Got next segment. Inline: false Aug 17 04:23:05.077065765 (202h56m45.808333395s) [upload=211167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:05.077069302 (202h56m45.808336922s) [upload=211167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:05.149595571 (202h56m45.880863191s) [seg_pos={0 0}, upload=211167]: Begin upload segment... Aug 17 04:23:05.149634344 (202h56m45.880901974s) [seg_pos={0 0}, upload=211167]: Done begin upload segment. Aug 17 04:23:05.149659451 (202h56m45.880927081s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.149699086 (202h56m45.880966716s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150081695 (202h56m45.881349325s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150110339 (202h56m45.88137799s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150390978 (202h56m45.881658608s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150446372 (202h56m45.881714002s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150693598 (202h56m45.881961228s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150755163 (202h56m45.882022793s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150983263 (202h56m45.882250893s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151079885 (202h56m45.882347515s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151274822 (202h56m45.882542452s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151365532 (202h56m45.882633163s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151584305 (202h56m45.882851935s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151603451 (202h56m45.882871081s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.152583125 (202h56m45.883850755s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.153130696 (202h56m45.884398326s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154141749 (202h56m45.88540938s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154561108 (202h56m45.885828738s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.222341082 (202h56m45.953608762s) [upload=211167]: Next returned nil segment Aug 17 04:23:05.222354026 (202h56m45.953621646s) [upload=211167]: Waiting for error group managing segments... Aug 17 04:23:05.230308020 (202h56m45.96157565s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.230754350 (202h56m45.96202198s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231421757 (202h56m45.962689387s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231643444 (202h56m45.962911074s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.296297613 (202h56m46.027565233s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.346576712 (202h56m46.077844332s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.494438107 (202h56m46.225705727s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.563775022 (202h56m46.295042642s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.674589117 (202h56m46.405856747s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.747386525 (202h56m46.478654145s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:24:05.061032967 (202h57m45.792300617s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:104.36.82.60:4351): protocol: rpc: dial tcp 104.36.82.60:4351: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061097619 (202h57m45.79236527s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: rpc: dial tcp 104.36.82.38:4696: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061148564 (202h57m45.792416215s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: rpc: dial tcp 104.36.82.17:4333: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061255446 (202h57m45.792523106s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12Nh9nHAnKWgu1ggCAZ7WcUadbvsPqAgbqzGrBnSaDMqDhquQ4R, address:104.36.82.22:4514): protocol: rpc: dial tcp 104.36.82.22:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061322663 (202h57m45.792590312s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, address:104.36.82.23:4446): protocol: write tcp 5.161.89.252:28480->104.36.82.23:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061363470 (202h57m45.79263112s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: rpc: dial tcp 104.36.82.2:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061387064 (202h57m45.792654714s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u, address:104.36.82.5:4864): protocol: rpc: dial tcp 104.36.82.5:4864: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061424123 (202h57m45.792691784s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA, address:104.36.82.23:4461): protocol: write tcp 5.161.89.252:18768->104.36.82.23:4461: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061453258 (202h57m45.792720908s) [seg_pos={0 0}, upload=211167]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:24:05.085107047 (202h57m45.816374697s) [upload=211167]: Done uploading segments... Aug 17 04:24:05.085125321 (202h57m45.816392971s) [upload=211167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 447.006564ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X.XX...............] duration: 602.573834ms FAIL (2025-08-17 04:34:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 602.594383ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.4465263s FAIL (2025-08-17 04:34:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.446543281s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................................XXXXXX.............] duration: 345.55712ms FAIL (2025-08-17 04:33:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:07.365016247 (202h56m48.096283887s) [upload=211168]: Starting upload Aug 17 04:23:07.365113880 (202h56m48.09638152s) [upload=211168]: Done starting upload Aug 17 04:23:07.365185295 (202h56m48.096452915s) [upload=211168]: Uploading segments... Aug 17 04:23:07.365444052 (202h56m48.096711693s) [upload=211168]: Got next segment. Inline: false Aug 17 04:23:07.365496200 (202h56m48.09676384s) [upload=211168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:07.365506590 (202h56m48.096774229s) [upload=211168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:07.391532593 (202h56m48.122800213s) [seg_pos={0 0}, upload=211168]: Begin upload segment... Aug 17 04:23:07.391561247 (202h56m48.122828867s) [seg_pos={0 0}, upload=211168]: Done begin upload segment. Aug 17 04:23:07.391580673 (202h56m48.122848294s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391635377 (202h56m48.122903017s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391895125 (202h56m48.123162745s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392097867 (202h56m48.123365508s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392117784 (202h56m48.123385405s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392313993 (202h56m48.123581614s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392395668 (202h56m48.123663308s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392493893 (202h56m48.123761513s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392681466 (202h56m48.123949106s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392702645 (202h56m48.123970275s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392977794 (202h56m48.124245414s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392988844 (202h56m48.124256475s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393185134 (202h56m48.124452754s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393273310 (202h56m48.12454094s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393400339 (202h56m48.124667959s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393572042 (202h56m48.124839683s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.461635871 (202h56m48.192903501s) [upload=211168]: Next returned nil segment Aug 17 04:23:07.461646972 (202h56m48.192914602s) [upload=211168]: Waiting for error group managing segments... Aug 17 04:23:07.469880881 (202h56m48.201148501s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470249144 (202h56m48.201516775s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470491772 (202h56m48.201759402s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470580539 (202h56m48.201848169s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470934566 (202h56m48.202202196s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471147235 (202h56m48.202414855s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471257373 (202h56m48.202525003s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471574630 (202h56m48.20284225s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471656044 (202h56m48.202923674s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.472781833 (202h56m48.204049453s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:37.353223355 (202h57m18.084491014s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353345955 (202h57m18.084613605s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353365382 (202h57m18.084633042s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353416519 (202h57m18.084684169s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353477142 (202h57m18.084744793s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 5.161.89.252:26914->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353487252 (202h57m18.084754901s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353530393 (202h57m18.084798093s) [seg_pos={0 0}, upload=211168]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:37.392615539 (202h57m18.123883199s) [upload=211168]: Done uploading segments... Aug 17 04:23:37.392649854 (202h57m18.123917504s) [upload=211168]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 345.576516ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................XXXX............] duration: 431.937349ms FAIL (2025-08-17 04:33:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== se Aug 17 04:23:28.620840633 (202h57m9.352108284s) [upload=211175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:28.620845953 (202h57m9.352113603s) [upload=211175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:28.646835158 (202h57m9.378102858s) [seg_pos={0 0}, upload=211175]: Begin upload segment... Aug 17 04:23:28.646936819 (202h57m9.378204468s) [seg_pos={0 0}, upload=211175]: Done begin upload segment. Aug 17 04:23:28.646989559 (202h57m9.378257218s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647051715 (202h57m9.378319375s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647662535 (202h57m9.378930186s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647827836 (202h57m9.379095487s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648134293 (202h57m9.379401954s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648314913 (202h57m9.379582573s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648543414 (202h57m9.379811063s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648834572 (202h57m9.380102222s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648992249 (202h57m9.380259908s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649335063 (202h57m9.380602713s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649507127 (202h57m9.380774777s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649830226 (202h57m9.381097876s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650003341 (202h57m9.381270992s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650256469 (202h57m9.381524128s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650434553 (202h57m9.381702213s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650709621 (202h57m9.381977282s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650857891 (202h57m9.38212554s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.651270246 (202h57m9.382537897s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.768946979 (202h57m9.500214629s) [upload=211175]: Next returned nil segment Aug 17 04:23:28.768974191 (202h57m9.500241841s) [upload=211175]: Waiting for error group managing segments... Aug 17 04:23:28.777572686 (202h57m9.508840347s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.778945461 (202h57m9.510213112s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.785896566 (202h57m9.517164206s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.793497595 (202h57m9.524765245s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.796220951 (202h57m9.527488592s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.799941495 (202h57m9.531209135s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.801171361 (202h57m9.532439001s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.802452863 (202h57m9.533720513s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.822657288 (202h57m9.553924938s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.874167434 (202h57m9.605435084s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.100609905 (202h57m9.831877555s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.216981541 (202h57m9.948249171s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:24:28.606052339 (202h58m9.337319999s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: rpc: dial tcp 104.36.82.44:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606141437 (202h58m9.337409087s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1AowWwVvxzmnW5qwbqbMBVbvNoigffjAN51LQp8e6XA9CZkeXL, address:104.36.82.3:4790): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606261834 (202h58m9.337529484s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:58906->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606338818 (202h58m9.337606469s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:104.36.82.23:4455): protocol: write tcp 5.161.89.252:44290->104.36.82.23:4455: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606369045 (202h58m9.337636696s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: rpc: dial tcp 104.36.82.60:4365: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606443665 (202h58m9.337711316s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, address:104.36.82.49:4003): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606480044 (202h58m9.337747704s) [seg_pos={0 0}, upload=211175]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 04:24:28.621101719 (202h58m9.352369369s) [upload=211175]: Done uploading segments... Aug 17 04:24:28.621117609 (202h58m9.35238526s) [upload=211175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 431.958711ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................XXXX...............] duration: 425.893008ms FAIL (2025-08-17 04:33:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 04:23:05.076746715 (202h56m45.808014345s) [upload=211167]: Starting upload Aug 17 04:23:05.076819060 (202h56m45.80808669s) [upload=211167]: Done starting upload Aug 17 04:23:05.076858426 (202h56m45.808126045s) [upload=211167]: Uploading segments... Aug 17 04:23:05.077035118 (202h56m45.808302748s) [upload=211167]: Got next segment. Inline: false Aug 17 04:23:05.077065765 (202h56m45.808333395s) [upload=211167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:05.077069302 (202h56m45.808336922s) [upload=211167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:05.149595571 (202h56m45.880863191s) [seg_pos={0 0}, upload=211167]: Begin upload segment... Aug 17 04:23:05.149634344 (202h56m45.880901974s) [seg_pos={0 0}, upload=211167]: Done begin upload segment. Aug 17 04:23:05.149659451 (202h56m45.880927081s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.149699086 (202h56m45.880966716s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150081695 (202h56m45.881349325s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150110339 (202h56m45.88137799s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150390978 (202h56m45.881658608s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150446372 (202h56m45.881714002s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150693598 (202h56m45.881961228s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150755163 (202h56m45.882022793s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.150983263 (202h56m45.882250893s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151079885 (202h56m45.882347515s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151274822 (202h56m45.882542452s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151365532 (202h56m45.882633163s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151584305 (202h56m45.882851935s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.151603451 (202h56m45.882871081s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.152583125 (202h56m45.883850755s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.153130696 (202h56m45.884398326s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154141749 (202h56m45.88540938s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.154561108 (202h56m45.885828738s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Uploading piece... Aug 17 04:23:05.222341082 (202h56m45.953608762s) [upload=211167]: Next returned nil segment Aug 17 04:23:05.222354026 (202h56m45.953621646s) [upload=211167]: Waiting for error group managing segments... Aug 17 04:23:05.230308020 (202h56m45.96157565s) [noise=true , address=45.91.116.7:4031 , piece_id=P7L222FZRXYRIACW4W7CIC5AEWMWRPTO2MIXEUAAZFIS567IJ5EA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.230754350 (202h56m45.96202198s) [noise=true , address=45.91.116.11:4147 , piece_id=O6V3FBZMK7DAIK4SFLZYQ2NVAQYJYG5JXMJJAHS6J2DW6ASNQMCQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231421757 (202h56m45.962689387s) [noise=true , address=77.90.31.11:4580 , piece_id=H5FGTVQYA7SJBS7WCGDLX5W5I6BQBXNQG3GKJLUM6PHXKAE6G5YQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.231643444 (202h56m45.962911074s) [noise=true , address=45.91.116.13:4181 , piece_id=SU23KT5A5GRXOG4Y6PHXGYK2EXLDPYBN5DB5Q27XOO5FJ2YWSQXA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.296297613 (202h56m46.027565233s) [noise=true , address=77.90.31.10:4523 , piece_id=2AV6APC7S3A37H5JTRIZEZAIOZ2TFL2YCLJRUXBHIP2PFEXUJI7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.346576712 (202h56m46.077844332s) [noise=true , address=77.90.31.19:4870 , piece_id=OLLFGZRKVZ6ABXS5Z6LQRI5HG2G356XSXE23LYMXFON5IRKOXIQQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.494438107 (202h56m46.225705727s) [noise=true , address=79.127.201.214:4309 , piece_id=HE4Y4EIXHL3HASHI5KDXHNT6S3YBNQ3MOCR7TGCM5MWBTAE7H6LA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.563775022 (202h56m46.295042642s) [noise=true , address=79.127.201.215:4350 , piece_id=J4FJ64ZNWN6TUH5KUE5Z6FRAE7Q64N4DXXFNXUIL3QNAFDCEBVUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.674589117 (202h56m46.405856747s) [noise=true , address=79.127.205.246:4283 , piece_id=AQKEHJQK5M6DHLVGT5LKJH6SAAJFYA2YEO7TTI7UBTJ7NHSBTWUA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:23:05.747386525 (202h56m46.478654145s) [noise=true , address=66.23.193.175:4559 , piece_id=QXEZFWPC3G25QKMLGXSPDZH4KIDVTZMK5AF7WY7IR2AOLQ37VFHA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: <nil> Aug 17 04:24:05.061032967 (202h57m45.792300617s) [noise=true , address=104.36.82.60:4351 , piece_id=JZSLHYLOWTUYL66X2CP6MCGLDMV3A5TYYMXDDSWXD5PFZ7TIGUPA, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg, address:104.36.82.60:4351): protocol: rpc: dial tcp 104.36.82.60:4351: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061097619 (202h57m45.79236527s) [noise=true , address=104.36.82.38:4696 , piece_id=NMDNAT7RVBIA6YQK24Z7XJQSOTIVOBIY5RIHK4EFWJK6N2ZL4UJQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: rpc: dial tcp 104.36.82.38:4696: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061148564 (202h57m45.792416215s) [noise=true , address=104.36.82.17:4333 , piece_id=AAKCYXF7BIEVWEYXGFSSRQDCQNXLWYBLQLYNX3364RG4656IED7Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: rpc: dial tcp 104.36.82.17:4333: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061255446 (202h57m45.792523106s) [noise=true , address=104.36.82.22:4514 , piece_id=73OAAQLS545E64H6VE53NXTAPBX4XJXPZN2EEX53EFUSKIBFTXLQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12Nh9nHAnKWgu1ggCAZ7WcUadbvsPqAgbqzGrBnSaDMqDhquQ4R, address:104.36.82.22:4514): protocol: rpc: dial tcp 104.36.82.22:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061322663 (202h57m45.792590312s) [noise=true , address=104.36.82.23:4446 , piece_id=VSLMTGTUUZSEP6X7SCPALTJFOA7IJJCIKLJHK6TIYR6VJQLHKTOQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, address:104.36.82.23:4446): protocol: write tcp 5.161.89.252:28480->104.36.82.23:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061363470 (202h57m45.79263112s) [noise=true , address=104.36.82.2:4762 , piece_id=CMIP2M7AMYAMK7MIDZU5XJRKNVG5ELS7DJBHP73FD2HQHOF2XRTQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12r1mav9KnS7y2V5XxeDexYM4Gv9JcfmqNbML5pwSEyzCZFvJra, address:104.36.82.2:4762): protocol: rpc: dial tcp 104.36.82.2:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061387064 (202h57m45.792654714s) [noise=true , address=104.36.82.5:4864 , piece_id=QU3BHU64HCXUVPBMCEBIYAWX72Q7SGM2I5YF46GNNDM5LJOD542Q, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u, address:104.36.82.5:4864): protocol: rpc: dial tcp 104.36.82.5:4864: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061424123 (202h57m45.792691784s) [noise=true , address=104.36.82.23:4461 , piece_id=JWAHLP4YYBWYQUWEQC5EHJZZZPVYOUTGC2TS5LNXP4SG7EKJFYYQ, seg_pos={0 0}, upload=211167]: Done uploading piece. err: ecclient: upload failed (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA, address:104.36.82.23:4461): protocol: write tcp 5.161.89.252:18768->104.36.82.23:4461: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:05.061453258 (202h57m45.792720908s) [seg_pos={0 0}, upload=211167]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:24:05.085107047 (202h57m45.816374697s) [upload=211167]: Done uploading segments... Aug 17 04:24:05.085125321 (202h57m45.816392971s) [upload=211167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 426.026328ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 1.590789159s FAIL (2025-08-17 04:33:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.590814407s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X.XX..............] duration: 405.354083ms FAIL (2025-08-17 04:33:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 405.370003ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................................XXXXXX............] duration: 428.916988ms FAIL (2025-08-17 04:32:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:07.365016247 (202h56m48.096283887s) [upload=211168]: Starting upload Aug 17 04:23:07.365113880 (202h56m48.09638152s) [upload=211168]: Done starting upload Aug 17 04:23:07.365185295 (202h56m48.096452915s) [upload=211168]: Uploading segments... Aug 17 04:23:07.365444052 (202h56m48.096711693s) [upload=211168]: Got next segment. Inline: false Aug 17 04:23:07.365496200 (202h56m48.09676384s) [upload=211168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:07.365506590 (202h56m48.096774229s) [upload=211168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:07.391532593 (202h56m48.122800213s) [seg_pos={0 0}, upload=211168]: Begin upload segment... Aug 17 04:23:07.391561247 (202h56m48.122828867s) [seg_pos={0 0}, upload=211168]: Done begin upload segment. Aug 17 04:23:07.391580673 (202h56m48.122848294s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391635377 (202h56m48.122903017s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.391895125 (202h56m48.123162745s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392097867 (202h56m48.123365508s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392117784 (202h56m48.123385405s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392313993 (202h56m48.123581614s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392395668 (202h56m48.123663308s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392493893 (202h56m48.123761513s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392681466 (202h56m48.123949106s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392702645 (202h56m48.123970275s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392977794 (202h56m48.124245414s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.392988844 (202h56m48.124256475s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393185134 (202h56m48.124452754s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393273310 (202h56m48.12454094s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393400339 (202h56m48.124667959s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.393572042 (202h56m48.124839683s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Uploading piece... Aug 17 04:23:07.461635871 (202h56m48.192903501s) [upload=211168]: Next returned nil segment Aug 17 04:23:07.461646972 (202h56m48.192914602s) [upload=211168]: Waiting for error group managing segments... Aug 17 04:23:07.469880881 (202h56m48.201148501s) [noise=true , address=45.91.116.4:4938 , piece_id=F3AVEZ2YIU4EJZCAGSWZGJPNJ7KNSRGRHMNMLPE7XQLMXWGL62XQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470249144 (202h56m48.201516775s) [noise=true , address=45.91.116.13:4154 , piece_id=NGDMGFLNRFPCKKQR72KBGJJXOTE5CV2W6OEQAWRJKCTENPUH3T5A, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470491772 (202h56m48.201759402s) [noise=true , address=45.91.116.7:4009 , piece_id=36E4LB3QDRVIJ7V3CFTBEU7XVJNMHPXMO3W4U2EWNRRKPQY6RNBQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470580539 (202h56m48.201848169s) [noise=true , address=45.91.116.4:4937 , piece_id=BEYPSNQUECATWBTX4JUM6NP2BBFXQSYVFBNMJFO5O4UNHRLTZLJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.470934566 (202h56m48.202202196s) [noise=true , address=45.91.116.14:4189 , piece_id=JRIG76HQYCBKFQ67GEXATVH3T5C5I6VLJWNU6XNSKRR5YIAF6UJA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471147235 (202h56m48.202414855s) [noise=true , address=45.91.116.10:4081 , piece_id=F4SEIIXTTLFASZHDJBYUBDYLJMAZC6K7MEAIPD3WAQ35UHAEAZMQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471257373 (202h56m48.202525003s) [noise=true , address=45.91.116.5:4298 , piece_id=SO6VVAFXQBPJMLOQOEFGGC4D6ICGBUK7BVMKHS4D3HYHCQWABJXA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471574630 (202h56m48.20284225s) [noise=true , address=45.91.116.7:4010 , piece_id=6RWG7YB22HSREMJA44RC43SBH3TPQ3F3EDKAC2JULNK2SWSBWNEQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.471656044 (202h56m48.202923674s) [noise=true , address=45.91.116.11:4117 , piece_id=OY5KNPDA5EI6OIMR5DBQ7IHSEJQK777NPIKWOZI3RZH7NIF7GWAQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:07.472781833 (202h56m48.204049453s) [noise=true , address=77.90.31.19:4841 , piece_id=L7TJCCUKL4RISGHZFGOFLKVNKJ3LK2PAAFWQCRJAV7N5VTNPSWDQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: <nil> Aug 17 04:23:37.353223355 (202h57m18.084491014s) [noise=true , address=104.36.82.4:4803 , piece_id=TV3P4HUUHIXIJ7P2ZONBQPAJAY3UZVVLFF4P7TXAZEU47Y6RJ4QQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353345955 (202h57m18.084613605s) [noise=true , address=104.36.82.5:4839 , piece_id=5C6WOWRH4JAPDZFHZYVT7L5GMKU2MYQGWQLIVRUUJJHOOD73HFFA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353365382 (202h57m18.084633042s) [noise=true , address=104.36.82.12:4151 , piece_id=3Q6OWEBKFSXBCK5AJ6NHFUIO2GXBO3A6QX7X5T2VNQK42KI5KLNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353416519 (202h57m18.084684169s) [noise=true , address=104.36.82.2:4730 , piece_id=NPIC6AKOB2DBFDFK36KDDJANJZN7JFTISF6DI2K4OTCJIOABPK6Q, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353477142 (202h57m18.084744793s) [noise=true , address=104.36.82.16:4295 , piece_id=7BPNXEZ5JSW4T6DUZED3BAGJZLUAGN2PGDHSXLUV34AC7BM3OGYA, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 5.161.89.252:26914->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353487252 (202h57m18.084754901s) [noise=true , address=104.36.82.8:4969 , piece_id=ZP65DV3U7BGQ4OHGXJWB6NSWJBIZIEFLERF7BNBIMTGH5YQWBTNQ, seg_pos={0 0}, upload=211168]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:37.353530393 (202h57m18.084798093s) [seg_pos={0 0}, upload=211168]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:37.392615539 (202h57m18.123883199s) [upload=211168]: Done uploading segments... Aug 17 04:23:37.392649854 (202h57m18.123917504s) [upload=211168]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 428.937005ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.554110867s FAIL (2025-08-17 04:32:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.554134411s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................XXXX...........] duration: 655.555814ms FAIL (2025-08-17 04:32:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== se Aug 17 04:23:28.620840633 (202h57m9.352108284s) [upload=211175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:28.620845953 (202h57m9.352113603s) [upload=211175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:28.646835158 (202h57m9.378102858s) [seg_pos={0 0}, upload=211175]: Begin upload segment... Aug 17 04:23:28.646936819 (202h57m9.378204468s) [seg_pos={0 0}, upload=211175]: Done begin upload segment. Aug 17 04:23:28.646989559 (202h57m9.378257218s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647051715 (202h57m9.378319375s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647662535 (202h57m9.378930186s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.647827836 (202h57m9.379095487s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648134293 (202h57m9.379401954s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648314913 (202h57m9.379582573s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648543414 (202h57m9.379811063s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648834572 (202h57m9.380102222s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.648992249 (202h57m9.380259908s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649335063 (202h57m9.380602713s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649507127 (202h57m9.380774777s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.649830226 (202h57m9.381097876s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650003341 (202h57m9.381270992s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650256469 (202h57m9.381524128s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650434553 (202h57m9.381702213s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650709621 (202h57m9.381977282s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.650857891 (202h57m9.38212554s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.651270246 (202h57m9.382537897s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Uploading piece... Aug 17 04:23:28.768946979 (202h57m9.500214629s) [upload=211175]: Next returned nil segment Aug 17 04:23:28.768974191 (202h57m9.500241841s) [upload=211175]: Waiting for error group managing segments... Aug 17 04:23:28.777572686 (202h57m9.508840347s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.778945461 (202h57m9.510213112s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.785896566 (202h57m9.517164206s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.793497595 (202h57m9.524765245s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.796220951 (202h57m9.527488592s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.799941495 (202h57m9.531209135s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.801171361 (202h57m9.532439001s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.802452863 (202h57m9.533720513s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.822657288 (202h57m9.553924938s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:28.874167434 (202h57m9.605435084s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.100609905 (202h57m9.831877555s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:23:29.216981541 (202h57m9.948249171s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil> Aug 17 04:24:28.606052339 (202h58m9.337319999s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: rpc: dial tcp 104.36.82.44:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606141437 (202h58m9.337409087s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1AowWwVvxzmnW5qwbqbMBVbvNoigffjAN51LQp8e6XA9CZkeXL, address:104.36.82.3:4790): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606261834 (202h58m9.337529484s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:58906->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606338818 (202h58m9.337606469s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:104.36.82.23:4455): protocol: write tcp 5.161.89.252:44290->104.36.82.23:4455: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606369045 (202h58m9.337636696s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: rpc: dial tcp 104.36.82.60:4365: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:24:28.606443665 (202h58m9.337711316s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, address:104.36.82.49:4003): protocol: context deadline exceeded; context deadline exceeded Aug 17 04:24:28.606480044 (202h58m9.337747704s) [seg_pos={0 0}, upload=211175]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 17 04:24:28.621101719 (202h58m9.352369369s) [upload=211175]: Done uploading segments... Aug 17 04:24:28.621117609 (202h58m9.35238526s) [upload=211175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 655.581414ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X.XX.............] duration: 496.419426ms FAIL (2025-08-17 04:32:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 496.443862ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)