Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X....................................X.X.X..........] duration: 655.232106ms FAIL (2025-08-19 17:31:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 19 17:24:12.613365846 (263h57m53.344633476s) [upload=274521]: Starting upload Aug 19 17:24:12.613406413 (263h57m53.344674043s) [upload=274521]: Done starting upload Aug 19 17:24:12.613493606 (263h57m53.344761226s) [upload=274521]: Uploading segments... Aug 19 17:24:12.613734899 (263h57m53.345002529s) [upload=274521]: Got next segment. Inline: false Aug 19 17:24:12.613764896 (263h57m53.345032526s) [upload=274521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:12.613769284 (263h57m53.345036914s) [upload=274521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:12.651951109 (263h57m53.383218739s) [seg_pos={0 0}, upload=274521]: Begin upload segment... Aug 19 17:24:12.651990573 (263h57m53.383258193s) [seg_pos={0 0}, upload=274521]: Done begin upload segment. Aug 19 17:24:12.652011803 (263h57m53.383279433s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652035357 (263h57m53.383302987s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652327476 (263h57m53.383595106s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652383030 (263h57m53.38365066s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652565463 (263h57m53.383833093s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652676883 (263h57m53.383944513s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652812508 (263h57m53.384080138s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652949715 (263h57m53.384217345s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653091241 (263h57m53.384358871s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653251271 (263h57m53.384518892s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653305264 (263h57m53.384572884s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653509577 (263h57m53.384777207s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653518354 (263h57m53.384785984s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653743006 (263h57m53.385010626s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653788721 (263h57m53.385056351s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653985512 (263h57m53.385253132s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654086841 (263h57m53.385354471s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654373080 (263h57m53.3856407s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.735474331 (263h57m53.466741961s) [upload=274521]: Next returned nil segment Aug 19 17:24:12.735486304 (263h57m53.466753934s) [upload=274521]: Waiting for error group managing segments... Aug 19 17:24:12.756883830 (263h57m53.48815145s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.766603935 (263h57m53.497871585s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.809581250 (263h57m53.540859941s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.908061427 (263h57m53.639329057s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.147463951 (263h57m53.878731601s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.364439955 (263h57m54.095707575s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:51.706547376 (263h58m32.437815006s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:52.370283683 (263h58m33.101551303s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.816229069 (263h58m36.547496699s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.944464809 (263h58m36.675732439s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.083334942 (263h58m48.814602572s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.405488480 (263h58m49.136756101s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.160489383 (263h58m52.891757013s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.595037822 (263h58m53.326305452s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595089570 (263h58m53.326357189s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 5.161.89.252:56044->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595102303 (263h58m53.326369933s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: rpc: dial tcp 45.91.116.4:4950: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595112262 (263h58m53.326379892s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 5.161.89.252:9456->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595145234 (263h58m53.326412864s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1gnJixpV2uZqEMTxi22y1EAzKCHXaQeBWakhT6AcEkrENWT8CT, address:45.91.116.13:4182): protocol: write tcp 5.161.89.252:9502->45.91.116.13:4182: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595163289 (263h58m53.326430909s) [seg_pos={0 0}, upload=274521]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 19 17:25:12.636371896 (263h58m53.367639527s) [upload=274521]: Done uploading segments... Aug 19 17:25:12.636391424 (263h58m53.367659054s) [upload=274521]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 655.254538ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X....X.........] duration: 374.467537ms FAIL (2025-08-19 17:31:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== [seg_pos={0 0}, upload=274520]: Done begin upload segment. Aug 19 17:24:12.024531859 (263h57m52.755799509s) [noise=true , address=104.36.82.9:4007 , piece_id=YEAQAGKXBODCVNIAG2S7TT5MP3TKTC5PAWGA7CIIBEW5DYS7UQCQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024547178 (263h57m52.755814828s) [noise=true , address=45.91.116.8:4045 , piece_id=X2HZK7QLI7RRZMF6WYSHK4TYUCPWMF7XRY6KVWKG7SOMGZK4HZXA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024953521 (263h57m52.756221171s) [noise=true , address=77.90.31.11:4554 , piece_id=OQYJJPQRLA64VFXA5VBWDKJ73Q2YMLTRW3UXJVNBQ3BLCZ55TZ5Q, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024997523 (263h57m52.756265174s) [noise=true , address=45.91.116.6:4973 , piece_id=MP533ZUEZDRXYTQPWKM5URFX6BB3WPGAPFHFMUJQVNBZFPTJHICA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025367439 (263h57m52.756635089s) [noise=true , address=77.90.31.19:4841 , piece_id=MY2CTXR6M4Z7JDC5BE3HENPMUOXXSVX22PNG236QOH5F3552PFCQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025414347 (263h57m52.756681997s) [noise=true , address=104.36.82.2:4729 , piece_id=LY2SWRDWXQHES5FTAF5CP6KCNVCY6FHIBKJNLOCEFYSBNBPIPNIA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025761920 (263h57m52.75702958s) [noise=true , address=104.36.82.7:4909 , piece_id=QB6Z7VE6VFC5L3BNINLJTO53YACDUGNBL6U7WUHKSG5HAFVSXAJQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025888298 (263h57m52.757155948s) [noise=true , address=77.90.31.14:4661 , piece_id=WDR237S6GBHWJGAO2L7M5YASTXX6QOLMIMDZM2D57PQHZA7BYB2Q, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026138559 (263h57m52.757406208s) [noise=true , address=77.90.31.12:4592 , piece_id=LYEZMNLTZVT4UQHZL53CGXZXPHLPJPO34EMLPPRARNVULCOQPXKQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026283681 (263h57m52.75755133s) [noise=true , address=104.36.82.5:4837 , piece_id=AM2AXREOT4ZII77PAWUE76BM6AR4YGI3XF5CGUWRKEE5R5FTDIFA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026491962 (263h57m52.757759613s) [noise=true , address=45.91.116.14:4189 , piece_id=QVER7IKF6T66GHOKDYVUVAYBF4OWORQV3IPKXKTREBYXOLKUI6HA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026640120 (263h57m52.757907771s) [noise=true , address=45.91.116.14:4190 , piece_id=3AZCPPFTZFZ7ANINXPAIR433CKJUXEVDB7FYJIQDLHMYIMRFTGKQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026842701 (263h57m52.758110352s) [noise=true , address=77.90.31.13:4628 , piece_id=B2BESUZXB37NOFSOEUPCUXX3AMEZ44S62HIKN7K4P2FD65CPEDPQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026995178 (263h57m52.758262828s) [noise=true , address=104.36.82.7:4912 , piece_id=ADZS3KGOOF4HPEYRXYTTS7WHDRKRBGRIKUKCORRSAENYPHXMEGHA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.027210582 (263h57m52.758478232s) [noise=true , address=104.36.82.13:4185 , piece_id=G6V7LQWJ3UC4PMMPBZWE67KWQXQENEEO4O4763VEGVGEVIDJ5FEA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.027858310 (263h57m52.75912596s) [noise=true , address=45.91.116.4:4939 , piece_id=BTZOJWID4G2ADAZKPQ6PW7SMV42R2FM756KFMCBSBAMFMJDFULWQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.143746757 (263h57m52.875014407s) [upload=274520]: Next returned nil segment Aug 19 17:24:12.143775502 (263h57m52.875043152s) [upload=274520]: Waiting for error group managing segments... Aug 19 17:24:12.157847553 (263h57m52.889115213s) [noise=true , address=104.36.82.13:4185 , piece_id=G6V7LQWJ3UC4PMMPBZWE67KWQXQENEEO4O4763VEGVGEVIDJ5FEA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.158547338 (263h57m52.889814989s) [noise=true , address=104.36.82.5:4837 , piece_id=AM2AXREOT4ZII77PAWUE76BM6AR4YGI3XF5CGUWRKEE5R5FTDIFA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.158808157 (263h57m52.890075818s) [noise=true , address=104.36.82.2:4729 , piece_id=LY2SWRDWXQHES5FTAF5CP6KCNVCY6FHIBKJNLOCEFYSBNBPIPNIA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.159159038 (263h57m52.890426688s) [noise=true , address=104.36.82.7:4912 , piece_id=ADZS3KGOOF4HPEYRXYTTS7WHDRKRBGRIKUKCORRSAENYPHXMEGHA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.160381044 (263h57m52.891648694s) [noise=true , address=104.36.82.7:4909 , piece_id=QB6Z7VE6VFC5L3BNINLJTO53YACDUGNBL6U7WUHKSG5HAFVSXAJQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.163470309 (263h57m52.894737959s) [noise=true , address=104.36.82.9:4007 , piece_id=YEAQAGKXBODCVNIAG2S7TT5MP3TKTC5PAWGA7CIIBEW5DYS7UQCQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:41.979520414 (263h58m22.710788064s) [noise=true , address=77.90.31.13:4628 , piece_id=B2BESUZXB37NOFSOEUPCUXX3AMEZ44S62HIKN7K4P2FD65CPEDPQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 5.161.89.252:36766->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980102297 (263h58m22.711369958s) [noise=true , address=45.91.116.14:4190 , piece_id=3AZCPPFTZFZ7ANINXPAIR433CKJUXEVDB7FYJIQDLHMYIMRFTGKQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:21646->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980158464 (263h58m22.711426114s) [noise=true , address=45.91.116.4:4939 , piece_id=BTZOJWID4G2ADAZKPQ6PW7SMV42R2FM756KFMCBSBAMFMJDFULWQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 5.161.89.252:51578->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980204240 (263h58m22.71147189s) [noise=true , address=77.90.31.11:4554 , piece_id=OQYJJPQRLA64VFXA5VBWDKJ73Q2YMLTRW3UXJVNBQ3BLCZ55TZ5Q, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 5.161.89.252:9778->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980249345 (263h58m22.711517004s) [noise=true , address=77.90.31.14:4661 , piece_id=WDR237S6GBHWJGAO2L7M5YASTXX6QOLMIMDZM2D57PQHZA7BYB2Q, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:22472->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980292254 (263h58m22.711559905s) [noise=true , address=45.91.116.14:4189 , piece_id=QVER7IKF6T66GHOKDYVUVAYBF4OWORQV3IPKXKTREBYXOLKUI6HA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: read tcp 5.161.89.252:34124->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980365091 (263h58m22.711632752s) [noise=true , address=45.91.116.8:4045 , piece_id=X2HZK7QLI7RRZMF6WYSHK4TYUCPWMF7XRY6KVWKG7SOMGZK4HZXA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: write tcp 5.161.89.252:8238->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980403213 (263h58m22.711670863s) [noise=true , address=77.90.31.19:4841 , piece_id=MY2CTXR6M4Z7JDC5BE3HENPMUOXXSVX22PNG236QOH5F3552PFCQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980547475 (263h58m22.711815124s) [noise=true , address=45.91.116.6:4973 , piece_id=MP533ZUEZDRXYTQPWKM5URFX6BB3WPGAPFHFMUJQVNBZFPTJHICA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 5.161.89.252:29404->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980632083 (263h58m22.711899743s) [noise=true , address=77.90.31.12:4592 , piece_id=LYEZMNLTZVT4UQHZL53CGXZXPHLPJPO34EMLPPRARNVULCOQPXKQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980736129 (263h58m22.712003779s) [seg_pos={0 0}, upload=274520]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:41.997898729 (263h58m22.729166379s) [upload=274520]: Done uploading segments... Aug 19 17:24:41.997919457 (263h58m22.729187108s) [upload=274520]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 374.497222ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X...X.........] duration: 595.274885ms FAIL (2025-08-19 17:31:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 595.299121ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............................X...........X..................] duration: 181.257639ms FAIL (2025-08-19 17:31:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:16:50.148435305 (263h50m30.879702956s) [upload=274393]: Starting upload Aug 19 17:16:50.148525875 (263h50m30.879793526s) [upload=274393]: Done starting upload Aug 19 17:16:50.148615374 (263h50m30.879883023s) [upload=274393]: Uploading segments... Aug 19 17:16:50.148667010 (263h50m30.879934661s) [upload=274393]: Got next segment. Inline: true Aug 19 17:16:50.148674014 (263h50m30.879941663s) [upload=274393]: Waiting for error group managing segments... Aug 19 17:16:50.148720332 (263h50m30.879987981s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:16:50.148725801 (263h50m30.879993452s) [upload=274393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:16:50.148729298 (263h50m30.879996948s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:16:50.221372078 (263h50m30.952639728s) [upload=274393]: Done uploading segments... Aug 19 17:16:50.221390462 (263h50m30.952658113s) [upload=274393]: Upload finished. err: metaclient: Too Many Requests Duration:* 181.277918ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...........X.................] duration: 141.92748ms FAIL (2025-08-19 17:30:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:16:50.148435305 (263h50m30.879702956s) [upload=274393]: Starting upload Aug 19 17:16:50.148525875 (263h50m30.879793526s) [upload=274393]: Done starting upload Aug 19 17:16:50.148615374 (263h50m30.879883023s) [upload=274393]: Uploading segments... Aug 19 17:16:50.148667010 (263h50m30.879934661s) [upload=274393]: Got next segment. Inline: true Aug 19 17:16:50.148674014 (263h50m30.879941663s) [upload=274393]: Waiting for error group managing segments... Aug 19 17:16:50.148720332 (263h50m30.879987981s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:16:50.148725801 (263h50m30.879993452s) [upload=274393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:16:50.148729298 (263h50m30.879996948s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:16:50.221372078 (263h50m30.952639728s) [upload=274393]: Done uploading segments... Aug 19 17:16:50.221390462 (263h50m30.952658113s) [upload=274393]: Upload finished. err: metaclient: Too Many Requests Duration:* 141.947767ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 631.921039ms FAIL (2025-08-19 17:30:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:20:30.724709450 (263h54m11.4559771s) [seg_pos={0 0}, upload=274457]: Begin upload segment... Aug 19 17:20:30.724753462 (263h54m11.456021113s) [seg_pos={0 0}, upload=274457]: Done begin upload segment. Aug 19 17:20:30.724781936 (263h54m11.456049586s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.724797054 (263h54m11.456064674s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725166569 (263h54m11.456434189s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725324816 (263h54m11.456592476s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725415496 (263h54m11.456683126s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725643935 (263h54m11.456911565s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725817342 (263h54m11.457084992s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725853730 (263h54m11.45712136s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725891440 (263h54m11.457159071s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726110873 (263h54m11.457378504s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726322932 (263h54m11.457590582s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726338811 (263h54m11.457606432s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726711522 (263h54m11.457979172s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727114529 (263h54m11.458382178s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727561679 (263h54m11.458829329s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728022655 (263h54m11.459290316s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728544566 (263h54m11.459812225s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728932415 (263h54m11.460200435s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.813701018 (263h54m11.544968628s) [upload=274457]: Next returned nil segment Aug 19 17:20:30.813711287 (263h54m11.544978897s) [upload=274457]: Waiting for error group managing segments... Aug 19 17:20:30.826872026 (263h54m11.558139646s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.832633702 (263h54m11.563901332s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.834038051 (263h54m11.565305681s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.844605367 (263h54m11.575872987s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.864222729 (263h54m11.59549035s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.866711214 (263h54m11.597978835s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.882824352 (263h54m11.614091972s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:31.250301081 (263h54m11.981568741s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:11.439696042 (263h54m52.170963692s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:13.585367789 (263h54m54.316635419s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:16.506635739 (263h54m57.237903369s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:22.880177884 (263h55m3.611445533s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:30.678773057 (263h55m11.410040716s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:14fc8QANkMVxxrHcXTWi1yyDhE2gz8BJNwRuvJNfX56TyQCWYZ, address:45.91.116.7:4032): protocol: rpc: dial tcp 45.91.116.7:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678847767 (263h55m11.410115416s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 5.161.89.252:32834->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678926845 (263h55m11.410194495s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: rpc: dial tcp 77.90.31.21:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678951581 (263h55m11.410219241s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, address:77.90.31.18:4829): protocol: rpc: dial tcp 77.90.31.18:4829: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678966439 (263h55m11.410234099s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: write tcp 5.161.89.252:42466->77.90.31.14:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679024038 (263h55m11.410291688s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7, address:45.91.116.10:4107): protocol: read tcp 5.161.89.252:39718->45.91.116.10:4107: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679071326 (263h55m11.410338977s) [seg_pos={0 0}, upload=274457]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 17:21:30.694027780 (263h55m11.425295431s) [upload=274457]: Done uploading segments... Aug 19 17:21:30.694046335 (263h55m11.425313986s) [upload=274457]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 631.949133ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.388329335s FAIL (2025-08-19 17:30:32 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 1.38834836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X....................................X.X.X.........] duration: 383.694905ms FAIL (2025-08-19 17:30:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 19 17:24:12.613365846 (263h57m53.344633476s) [upload=274521]: Starting upload Aug 19 17:24:12.613406413 (263h57m53.344674043s) [upload=274521]: Done starting upload Aug 19 17:24:12.613493606 (263h57m53.344761226s) [upload=274521]: Uploading segments... Aug 19 17:24:12.613734899 (263h57m53.345002529s) [upload=274521]: Got next segment. Inline: false Aug 19 17:24:12.613764896 (263h57m53.345032526s) [upload=274521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:12.613769284 (263h57m53.345036914s) [upload=274521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:12.651951109 (263h57m53.383218739s) [seg_pos={0 0}, upload=274521]: Begin upload segment... Aug 19 17:24:12.651990573 (263h57m53.383258193s) [seg_pos={0 0}, upload=274521]: Done begin upload segment. Aug 19 17:24:12.652011803 (263h57m53.383279433s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652035357 (263h57m53.383302987s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652327476 (263h57m53.383595106s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652383030 (263h57m53.38365066s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652565463 (263h57m53.383833093s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652676883 (263h57m53.383944513s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652812508 (263h57m53.384080138s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652949715 (263h57m53.384217345s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653091241 (263h57m53.384358871s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653251271 (263h57m53.384518892s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653305264 (263h57m53.384572884s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653509577 (263h57m53.384777207s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653518354 (263h57m53.384785984s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653743006 (263h57m53.385010626s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653788721 (263h57m53.385056351s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653985512 (263h57m53.385253132s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654086841 (263h57m53.385354471s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654373080 (263h57m53.3856407s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.735474331 (263h57m53.466741961s) [upload=274521]: Next returned nil segment Aug 19 17:24:12.735486304 (263h57m53.466753934s) [upload=274521]: Waiting for error group managing segments... Aug 19 17:24:12.756883830 (263h57m53.48815145s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.766603935 (263h57m53.497871585s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.809581250 (263h57m53.540859941s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.908061427 (263h57m53.639329057s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.147463951 (263h57m53.878731601s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.364439955 (263h57m54.095707575s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:51.706547376 (263h58m32.437815006s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:52.370283683 (263h58m33.101551303s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.816229069 (263h58m36.547496699s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.944464809 (263h58m36.675732439s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.083334942 (263h58m48.814602572s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.405488480 (263h58m49.136756101s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.160489383 (263h58m52.891757013s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.595037822 (263h58m53.326305452s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595089570 (263h58m53.326357189s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 5.161.89.252:56044->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595102303 (263h58m53.326369933s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: rpc: dial tcp 45.91.116.4:4950: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595112262 (263h58m53.326379892s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 5.161.89.252:9456->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595145234 (263h58m53.326412864s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1gnJixpV2uZqEMTxi22y1EAzKCHXaQeBWakhT6AcEkrENWT8CT, address:45.91.116.13:4182): protocol: write tcp 5.161.89.252:9502->45.91.116.13:4182: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595163289 (263h58m53.326430909s) [seg_pos={0 0}, upload=274521]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 19 17:25:12.636371896 (263h58m53.367639527s) [upload=274521]: Done uploading segments... Aug 19 17:25:12.636391424 (263h58m53.367659054s) [upload=274521]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 383.716046ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X....................................X.X.X........] duration: 366.206614ms FAIL (2025-08-19 17:30:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 19 17:24:12.613365846 (263h57m53.344633476s) [upload=274521]: Starting upload Aug 19 17:24:12.613406413 (263h57m53.344674043s) [upload=274521]: Done starting upload Aug 19 17:24:12.613493606 (263h57m53.344761226s) [upload=274521]: Uploading segments... Aug 19 17:24:12.613734899 (263h57m53.345002529s) [upload=274521]: Got next segment. Inline: false Aug 19 17:24:12.613764896 (263h57m53.345032526s) [upload=274521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:12.613769284 (263h57m53.345036914s) [upload=274521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:12.651951109 (263h57m53.383218739s) [seg_pos={0 0}, upload=274521]: Begin upload segment... Aug 19 17:24:12.651990573 (263h57m53.383258193s) [seg_pos={0 0}, upload=274521]: Done begin upload segment. Aug 19 17:24:12.652011803 (263h57m53.383279433s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652035357 (263h57m53.383302987s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652327476 (263h57m53.383595106s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652383030 (263h57m53.38365066s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652565463 (263h57m53.383833093s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652676883 (263h57m53.383944513s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652812508 (263h57m53.384080138s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652949715 (263h57m53.384217345s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653091241 (263h57m53.384358871s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653251271 (263h57m53.384518892s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653305264 (263h57m53.384572884s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653509577 (263h57m53.384777207s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653518354 (263h57m53.384785984s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653743006 (263h57m53.385010626s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653788721 (263h57m53.385056351s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653985512 (263h57m53.385253132s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654086841 (263h57m53.385354471s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654373080 (263h57m53.3856407s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.735474331 (263h57m53.466741961s) [upload=274521]: Next returned nil segment Aug 19 17:24:12.735486304 (263h57m53.466753934s) [upload=274521]: Waiting for error group managing segments... Aug 19 17:24:12.756883830 (263h57m53.48815145s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.766603935 (263h57m53.497871585s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.809581250 (263h57m53.540859941s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.908061427 (263h57m53.639329057s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.147463951 (263h57m53.878731601s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.364439955 (263h57m54.095707575s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:51.706547376 (263h58m32.437815006s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:52.370283683 (263h58m33.101551303s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.816229069 (263h58m36.547496699s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.944464809 (263h58m36.675732439s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.083334942 (263h58m48.814602572s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.405488480 (263h58m49.136756101s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.160489383 (263h58m52.891757013s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.595037822 (263h58m53.326305452s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595089570 (263h58m53.326357189s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 5.161.89.252:56044->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595102303 (263h58m53.326369933s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: rpc: dial tcp 45.91.116.4:4950: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595112262 (263h58m53.326379892s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 5.161.89.252:9456->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595145234 (263h58m53.326412864s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1gnJixpV2uZqEMTxi22y1EAzKCHXaQeBWakhT6AcEkrENWT8CT, address:45.91.116.13:4182): protocol: write tcp 5.161.89.252:9502->45.91.116.13:4182: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595163289 (263h58m53.326430909s) [seg_pos={0 0}, upload=274521]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 19 17:25:12.636371896 (263h58m53.367639527s) [upload=274521]: Done uploading segments... Aug 19 17:25:12.636391424 (263h58m53.367659054s) [upload=274521]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 366.235671ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..............................................X...X........] duration: 409.903221ms FAIL (2025-08-19 17:30:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 409.91964ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X....X........] duration: 377.680927ms FAIL (2025-08-19 17:30:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== [seg_pos={0 0}, upload=274520]: Done begin upload segment. Aug 19 17:24:12.024531859 (263h57m52.755799509s) [noise=true , address=104.36.82.9:4007 , piece_id=YEAQAGKXBODCVNIAG2S7TT5MP3TKTC5PAWGA7CIIBEW5DYS7UQCQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024547178 (263h57m52.755814828s) [noise=true , address=45.91.116.8:4045 , piece_id=X2HZK7QLI7RRZMF6WYSHK4TYUCPWMF7XRY6KVWKG7SOMGZK4HZXA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024953521 (263h57m52.756221171s) [noise=true , address=77.90.31.11:4554 , piece_id=OQYJJPQRLA64VFXA5VBWDKJ73Q2YMLTRW3UXJVNBQ3BLCZ55TZ5Q, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024997523 (263h57m52.756265174s) [noise=true , address=45.91.116.6:4973 , piece_id=MP533ZUEZDRXYTQPWKM5URFX6BB3WPGAPFHFMUJQVNBZFPTJHICA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025367439 (263h57m52.756635089s) [noise=true , address=77.90.31.19:4841 , piece_id=MY2CTXR6M4Z7JDC5BE3HENPMUOXXSVX22PNG236QOH5F3552PFCQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025414347 (263h57m52.756681997s) [noise=true , address=104.36.82.2:4729 , piece_id=LY2SWRDWXQHES5FTAF5CP6KCNVCY6FHIBKJNLOCEFYSBNBPIPNIA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025761920 (263h57m52.75702958s) [noise=true , address=104.36.82.7:4909 , piece_id=QB6Z7VE6VFC5L3BNINLJTO53YACDUGNBL6U7WUHKSG5HAFVSXAJQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025888298 (263h57m52.757155948s) [noise=true , address=77.90.31.14:4661 , piece_id=WDR237S6GBHWJGAO2L7M5YASTXX6QOLMIMDZM2D57PQHZA7BYB2Q, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026138559 (263h57m52.757406208s) [noise=true , address=77.90.31.12:4592 , piece_id=LYEZMNLTZVT4UQHZL53CGXZXPHLPJPO34EMLPPRARNVULCOQPXKQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026283681 (263h57m52.75755133s) [noise=true , address=104.36.82.5:4837 , piece_id=AM2AXREOT4ZII77PAWUE76BM6AR4YGI3XF5CGUWRKEE5R5FTDIFA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026491962 (263h57m52.757759613s) [noise=true , address=45.91.116.14:4189 , piece_id=QVER7IKF6T66GHOKDYVUVAYBF4OWORQV3IPKXKTREBYXOLKUI6HA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026640120 (263h57m52.757907771s) [noise=true , address=45.91.116.14:4190 , piece_id=3AZCPPFTZFZ7ANINXPAIR433CKJUXEVDB7FYJIQDLHMYIMRFTGKQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026842701 (263h57m52.758110352s) [noise=true , address=77.90.31.13:4628 , piece_id=B2BESUZXB37NOFSOEUPCUXX3AMEZ44S62HIKN7K4P2FD65CPEDPQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026995178 (263h57m52.758262828s) [noise=true , address=104.36.82.7:4912 , piece_id=ADZS3KGOOF4HPEYRXYTTS7WHDRKRBGRIKUKCORRSAENYPHXMEGHA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.027210582 (263h57m52.758478232s) [noise=true , address=104.36.82.13:4185 , piece_id=G6V7LQWJ3UC4PMMPBZWE67KWQXQENEEO4O4763VEGVGEVIDJ5FEA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.027858310 (263h57m52.75912596s) [noise=true , address=45.91.116.4:4939 , piece_id=BTZOJWID4G2ADAZKPQ6PW7SMV42R2FM756KFMCBSBAMFMJDFULWQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.143746757 (263h57m52.875014407s) [upload=274520]: Next returned nil segment Aug 19 17:24:12.143775502 (263h57m52.875043152s) [upload=274520]: Waiting for error group managing segments... Aug 19 17:24:12.157847553 (263h57m52.889115213s) [noise=true , address=104.36.82.13:4185 , piece_id=G6V7LQWJ3UC4PMMPBZWE67KWQXQENEEO4O4763VEGVGEVIDJ5FEA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.158547338 (263h57m52.889814989s) [noise=true , address=104.36.82.5:4837 , piece_id=AM2AXREOT4ZII77PAWUE76BM6AR4YGI3XF5CGUWRKEE5R5FTDIFA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.158808157 (263h57m52.890075818s) [noise=true , address=104.36.82.2:4729 , piece_id=LY2SWRDWXQHES5FTAF5CP6KCNVCY6FHIBKJNLOCEFYSBNBPIPNIA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.159159038 (263h57m52.890426688s) [noise=true , address=104.36.82.7:4912 , piece_id=ADZS3KGOOF4HPEYRXYTTS7WHDRKRBGRIKUKCORRSAENYPHXMEGHA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.160381044 (263h57m52.891648694s) [noise=true , address=104.36.82.7:4909 , piece_id=QB6Z7VE6VFC5L3BNINLJTO53YACDUGNBL6U7WUHKSG5HAFVSXAJQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.163470309 (263h57m52.894737959s) [noise=true , address=104.36.82.9:4007 , piece_id=YEAQAGKXBODCVNIAG2S7TT5MP3TKTC5PAWGA7CIIBEW5DYS7UQCQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:41.979520414 (263h58m22.710788064s) [noise=true , address=77.90.31.13:4628 , piece_id=B2BESUZXB37NOFSOEUPCUXX3AMEZ44S62HIKN7K4P2FD65CPEDPQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 5.161.89.252:36766->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980102297 (263h58m22.711369958s) [noise=true , address=45.91.116.14:4190 , piece_id=3AZCPPFTZFZ7ANINXPAIR433CKJUXEVDB7FYJIQDLHMYIMRFTGKQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:21646->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980158464 (263h58m22.711426114s) [noise=true , address=45.91.116.4:4939 , piece_id=BTZOJWID4G2ADAZKPQ6PW7SMV42R2FM756KFMCBSBAMFMJDFULWQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 5.161.89.252:51578->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980204240 (263h58m22.71147189s) [noise=true , address=77.90.31.11:4554 , piece_id=OQYJJPQRLA64VFXA5VBWDKJ73Q2YMLTRW3UXJVNBQ3BLCZ55TZ5Q, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 5.161.89.252:9778->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980249345 (263h58m22.711517004s) [noise=true , address=77.90.31.14:4661 , piece_id=WDR237S6GBHWJGAO2L7M5YASTXX6QOLMIMDZM2D57PQHZA7BYB2Q, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:22472->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980292254 (263h58m22.711559905s) [noise=true , address=45.91.116.14:4189 , piece_id=QVER7IKF6T66GHOKDYVUVAYBF4OWORQV3IPKXKTREBYXOLKUI6HA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: read tcp 5.161.89.252:34124->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980365091 (263h58m22.711632752s) [noise=true , address=45.91.116.8:4045 , piece_id=X2HZK7QLI7RRZMF6WYSHK4TYUCPWMF7XRY6KVWKG7SOMGZK4HZXA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: write tcp 5.161.89.252:8238->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980403213 (263h58m22.711670863s) [noise=true , address=77.90.31.19:4841 , piece_id=MY2CTXR6M4Z7JDC5BE3HENPMUOXXSVX22PNG236QOH5F3552PFCQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980547475 (263h58m22.711815124s) [noise=true , address=45.91.116.6:4973 , piece_id=MP533ZUEZDRXYTQPWKM5URFX6BB3WPGAPFHFMUJQVNBZFPTJHICA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 5.161.89.252:29404->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980632083 (263h58m22.711899743s) [noise=true , address=77.90.31.12:4592 , piece_id=LYEZMNLTZVT4UQHZL53CGXZXPHLPJPO34EMLPPRARNVULCOQPXKQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980736129 (263h58m22.712003779s) [seg_pos={0 0}, upload=274520]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:41.997898729 (263h58m22.729166379s) [upload=274520]: Done uploading segments... Aug 19 17:24:41.997919457 (263h58m22.729187108s) [upload=274520]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 377.707177ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...........X................] duration: 139.949192ms FAIL (2025-08-19 17:30:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:16:50.148435305 (263h50m30.879702956s) [upload=274393]: Starting upload Aug 19 17:16:50.148525875 (263h50m30.879793526s) [upload=274393]: Done starting upload Aug 19 17:16:50.148615374 (263h50m30.879883023s) [upload=274393]: Uploading segments... Aug 19 17:16:50.148667010 (263h50m30.879934661s) [upload=274393]: Got next segment. Inline: true Aug 19 17:16:50.148674014 (263h50m30.879941663s) [upload=274393]: Waiting for error group managing segments... Aug 19 17:16:50.148720332 (263h50m30.879987981s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:16:50.148725801 (263h50m30.879993452s) [upload=274393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:16:50.148729298 (263h50m30.879996948s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:16:50.221372078 (263h50m30.952639728s) [upload=274393]: Done uploading segments... Aug 19 17:16:50.221390462 (263h50m30.952658113s) [upload=274393]: Upload finished. err: metaclient: Too Many Requests Duration:* 139.985661ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 450.497411ms FAIL (2025-08-19 17:30:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:20:30.724709450 (263h54m11.4559771s) [seg_pos={0 0}, upload=274457]: Begin upload segment... Aug 19 17:20:30.724753462 (263h54m11.456021113s) [seg_pos={0 0}, upload=274457]: Done begin upload segment. Aug 19 17:20:30.724781936 (263h54m11.456049586s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.724797054 (263h54m11.456064674s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725166569 (263h54m11.456434189s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725324816 (263h54m11.456592476s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725415496 (263h54m11.456683126s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725643935 (263h54m11.456911565s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725817342 (263h54m11.457084992s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725853730 (263h54m11.45712136s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725891440 (263h54m11.457159071s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726110873 (263h54m11.457378504s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726322932 (263h54m11.457590582s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726338811 (263h54m11.457606432s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726711522 (263h54m11.457979172s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727114529 (263h54m11.458382178s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727561679 (263h54m11.458829329s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728022655 (263h54m11.459290316s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728544566 (263h54m11.459812225s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728932415 (263h54m11.460200435s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.813701018 (263h54m11.544968628s) [upload=274457]: Next returned nil segment Aug 19 17:20:30.813711287 (263h54m11.544978897s) [upload=274457]: Waiting for error group managing segments... Aug 19 17:20:30.826872026 (263h54m11.558139646s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.832633702 (263h54m11.563901332s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.834038051 (263h54m11.565305681s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.844605367 (263h54m11.575872987s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.864222729 (263h54m11.59549035s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.866711214 (263h54m11.597978835s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.882824352 (263h54m11.614091972s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:31.250301081 (263h54m11.981568741s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:11.439696042 (263h54m52.170963692s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:13.585367789 (263h54m54.316635419s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:16.506635739 (263h54m57.237903369s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:22.880177884 (263h55m3.611445533s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:30.678773057 (263h55m11.410040716s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:14fc8QANkMVxxrHcXTWi1yyDhE2gz8BJNwRuvJNfX56TyQCWYZ, address:45.91.116.7:4032): protocol: rpc: dial tcp 45.91.116.7:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678847767 (263h55m11.410115416s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 5.161.89.252:32834->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678926845 (263h55m11.410194495s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: rpc: dial tcp 77.90.31.21:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678951581 (263h55m11.410219241s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, address:77.90.31.18:4829): protocol: rpc: dial tcp 77.90.31.18:4829: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678966439 (263h55m11.410234099s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: write tcp 5.161.89.252:42466->77.90.31.14:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679024038 (263h55m11.410291688s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7, address:45.91.116.10:4107): protocol: read tcp 5.161.89.252:39718->45.91.116.10:4107: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679071326 (263h55m11.410338977s) [seg_pos={0 0}, upload=274457]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 17:21:30.694027780 (263h55m11.425295431s) [upload=274457]: Done uploading segments... Aug 19 17:21:30.694046335 (263h55m11.425313986s) [upload=274457]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 450.51781ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 416.780603ms FAIL (2025-08-19 17:29:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:20:30.724709450 (263h54m11.4559771s) [seg_pos={0 0}, upload=274457]: Begin upload segment... Aug 19 17:20:30.724753462 (263h54m11.456021113s) [seg_pos={0 0}, upload=274457]: Done begin upload segment. Aug 19 17:20:30.724781936 (263h54m11.456049586s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.724797054 (263h54m11.456064674s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725166569 (263h54m11.456434189s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725324816 (263h54m11.456592476s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725415496 (263h54m11.456683126s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725643935 (263h54m11.456911565s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725817342 (263h54m11.457084992s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725853730 (263h54m11.45712136s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725891440 (263h54m11.457159071s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726110873 (263h54m11.457378504s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726322932 (263h54m11.457590582s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726338811 (263h54m11.457606432s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726711522 (263h54m11.457979172s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727114529 (263h54m11.458382178s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727561679 (263h54m11.458829329s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728022655 (263h54m11.459290316s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728544566 (263h54m11.459812225s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728932415 (263h54m11.460200435s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.813701018 (263h54m11.544968628s) [upload=274457]: Next returned nil segment Aug 19 17:20:30.813711287 (263h54m11.544978897s) [upload=274457]: Waiting for error group managing segments... Aug 19 17:20:30.826872026 (263h54m11.558139646s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.832633702 (263h54m11.563901332s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.834038051 (263h54m11.565305681s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.844605367 (263h54m11.575872987s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.864222729 (263h54m11.59549035s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.866711214 (263h54m11.597978835s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.882824352 (263h54m11.614091972s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:31.250301081 (263h54m11.981568741s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:11.439696042 (263h54m52.170963692s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:13.585367789 (263h54m54.316635419s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:16.506635739 (263h54m57.237903369s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:22.880177884 (263h55m3.611445533s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:30.678773057 (263h55m11.410040716s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:14fc8QANkMVxxrHcXTWi1yyDhE2gz8BJNwRuvJNfX56TyQCWYZ, address:45.91.116.7:4032): protocol: rpc: dial tcp 45.91.116.7:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678847767 (263h55m11.410115416s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 5.161.89.252:32834->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678926845 (263h55m11.410194495s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: rpc: dial tcp 77.90.31.21:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678951581 (263h55m11.410219241s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, address:77.90.31.18:4829): protocol: rpc: dial tcp 77.90.31.18:4829: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678966439 (263h55m11.410234099s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: write tcp 5.161.89.252:42466->77.90.31.14:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679024038 (263h55m11.410291688s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7, address:45.91.116.10:4107): protocol: read tcp 5.161.89.252:39718->45.91.116.10:4107: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679071326 (263h55m11.410338977s) [seg_pos={0 0}, upload=274457]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 17:21:30.694027780 (263h55m11.425295431s) [upload=274457]: Done uploading segments... Aug 19 17:21:30.694046335 (263h55m11.425313986s) [upload=274457]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 416.81128ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................................X...........X...............] duration: 153.679083ms FAIL (2025-08-19 17:29:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:16:50.148435305 (263h50m30.879702956s) [upload=274393]: Starting upload Aug 19 17:16:50.148525875 (263h50m30.879793526s) [upload=274393]: Done starting upload Aug 19 17:16:50.148615374 (263h50m30.879883023s) [upload=274393]: Uploading segments... Aug 19 17:16:50.148667010 (263h50m30.879934661s) [upload=274393]: Got next segment. Inline: true Aug 19 17:16:50.148674014 (263h50m30.879941663s) [upload=274393]: Waiting for error group managing segments... Aug 19 17:16:50.148720332 (263h50m30.879987981s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:16:50.148725801 (263h50m30.879993452s) [upload=274393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:16:50.148729298 (263h50m30.879996948s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:16:50.221372078 (263h50m30.952639728s) [upload=274393]: Done uploading segments... Aug 19 17:16:50.221390462 (263h50m30.952658113s) [upload=274393]: Upload finished. err: metaclient: Too Many Requests Duration:* 153.702277ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.709999866s FAIL (2025-08-19 17:29:48 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 1.710016588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 473.086802ms FAIL (2025-08-19 17:29:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:20:30.724709450 (263h54m11.4559771s) [seg_pos={0 0}, upload=274457]: Begin upload segment... Aug 19 17:20:30.724753462 (263h54m11.456021113s) [seg_pos={0 0}, upload=274457]: Done begin upload segment. Aug 19 17:20:30.724781936 (263h54m11.456049586s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.724797054 (263h54m11.456064674s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725166569 (263h54m11.456434189s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725324816 (263h54m11.456592476s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725415496 (263h54m11.456683126s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725643935 (263h54m11.456911565s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725817342 (263h54m11.457084992s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725853730 (263h54m11.45712136s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725891440 (263h54m11.457159071s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726110873 (263h54m11.457378504s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726322932 (263h54m11.457590582s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726338811 (263h54m11.457606432s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726711522 (263h54m11.457979172s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727114529 (263h54m11.458382178s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727561679 (263h54m11.458829329s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728022655 (263h54m11.459290316s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728544566 (263h54m11.459812225s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728932415 (263h54m11.460200435s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.813701018 (263h54m11.544968628s) [upload=274457]: Next returned nil segment Aug 19 17:20:30.813711287 (263h54m11.544978897s) [upload=274457]: Waiting for error group managing segments... Aug 19 17:20:30.826872026 (263h54m11.558139646s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.832633702 (263h54m11.563901332s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.834038051 (263h54m11.565305681s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.844605367 (263h54m11.575872987s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.864222729 (263h54m11.59549035s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.866711214 (263h54m11.597978835s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.882824352 (263h54m11.614091972s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:31.250301081 (263h54m11.981568741s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:11.439696042 (263h54m52.170963692s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:13.585367789 (263h54m54.316635419s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:16.506635739 (263h54m57.237903369s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:22.880177884 (263h55m3.611445533s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:30.678773057 (263h55m11.410040716s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:14fc8QANkMVxxrHcXTWi1yyDhE2gz8BJNwRuvJNfX56TyQCWYZ, address:45.91.116.7:4032): protocol: rpc: dial tcp 45.91.116.7:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678847767 (263h55m11.410115416s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 5.161.89.252:32834->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678926845 (263h55m11.410194495s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: rpc: dial tcp 77.90.31.21:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678951581 (263h55m11.410219241s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, address:77.90.31.18:4829): protocol: rpc: dial tcp 77.90.31.18:4829: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678966439 (263h55m11.410234099s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: write tcp 5.161.89.252:42466->77.90.31.14:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679024038 (263h55m11.410291688s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7, address:45.91.116.10:4107): protocol: read tcp 5.161.89.252:39718->45.91.116.10:4107: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679071326 (263h55m11.410338977s) [seg_pos={0 0}, upload=274457]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 17:21:30.694027780 (263h55m11.425295431s) [upload=274457]: Done uploading segments... Aug 19 17:21:30.694046335 (263h55m11.425313986s) [upload=274457]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 473.1126ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X....................................X.X.X.......] duration: 447.515198ms FAIL (2025-08-19 17:29:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 19 17:24:12.613365846 (263h57m53.344633476s) [upload=274521]: Starting upload Aug 19 17:24:12.613406413 (263h57m53.344674043s) [upload=274521]: Done starting upload Aug 19 17:24:12.613493606 (263h57m53.344761226s) [upload=274521]: Uploading segments... Aug 19 17:24:12.613734899 (263h57m53.345002529s) [upload=274521]: Got next segment. Inline: false Aug 19 17:24:12.613764896 (263h57m53.345032526s) [upload=274521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:12.613769284 (263h57m53.345036914s) [upload=274521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:12.651951109 (263h57m53.383218739s) [seg_pos={0 0}, upload=274521]: Begin upload segment... Aug 19 17:24:12.651990573 (263h57m53.383258193s) [seg_pos={0 0}, upload=274521]: Done begin upload segment. Aug 19 17:24:12.652011803 (263h57m53.383279433s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652035357 (263h57m53.383302987s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652327476 (263h57m53.383595106s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652383030 (263h57m53.38365066s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652565463 (263h57m53.383833093s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652676883 (263h57m53.383944513s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652812508 (263h57m53.384080138s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652949715 (263h57m53.384217345s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653091241 (263h57m53.384358871s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653251271 (263h57m53.384518892s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653305264 (263h57m53.384572884s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653509577 (263h57m53.384777207s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653518354 (263h57m53.384785984s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653743006 (263h57m53.385010626s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653788721 (263h57m53.385056351s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653985512 (263h57m53.385253132s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654086841 (263h57m53.385354471s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654373080 (263h57m53.3856407s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.735474331 (263h57m53.466741961s) [upload=274521]: Next returned nil segment Aug 19 17:24:12.735486304 (263h57m53.466753934s) [upload=274521]: Waiting for error group managing segments... Aug 19 17:24:12.756883830 (263h57m53.48815145s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.766603935 (263h57m53.497871585s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.809581250 (263h57m53.540859941s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.908061427 (263h57m53.639329057s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.147463951 (263h57m53.878731601s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.364439955 (263h57m54.095707575s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:51.706547376 (263h58m32.437815006s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:52.370283683 (263h58m33.101551303s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.816229069 (263h58m36.547496699s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.944464809 (263h58m36.675732439s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.083334942 (263h58m48.814602572s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.405488480 (263h58m49.136756101s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.160489383 (263h58m52.891757013s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.595037822 (263h58m53.326305452s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595089570 (263h58m53.326357189s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 5.161.89.252:56044->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595102303 (263h58m53.326369933s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: rpc: dial tcp 45.91.116.4:4950: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595112262 (263h58m53.326379892s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 5.161.89.252:9456->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595145234 (263h58m53.326412864s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1gnJixpV2uZqEMTxi22y1EAzKCHXaQeBWakhT6AcEkrENWT8CT, address:45.91.116.13:4182): protocol: write tcp 5.161.89.252:9502->45.91.116.13:4182: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595163289 (263h58m53.326430909s) [seg_pos={0 0}, upload=274521]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 19 17:25:12.636371896 (263h58m53.367639527s) [upload=274521]: Done uploading segments... Aug 19 17:25:12.636391424 (263h58m53.367659054s) [upload=274521]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 447.541007ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XX..............................................X...X.......] duration: 428.861672ms FAIL (2025-08-19 17:29:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 428.87601ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..............................................X....X.......] duration: 389.34822ms FAIL (2025-08-19 17:29:23 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== [seg_pos={0 0}, upload=274520]: Done begin upload segment. Aug 19 17:24:12.024531859 (263h57m52.755799509s) [noise=true , address=104.36.82.9:4007 , piece_id=YEAQAGKXBODCVNIAG2S7TT5MP3TKTC5PAWGA7CIIBEW5DYS7UQCQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024547178 (263h57m52.755814828s) [noise=true , address=45.91.116.8:4045 , piece_id=X2HZK7QLI7RRZMF6WYSHK4TYUCPWMF7XRY6KVWKG7SOMGZK4HZXA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024953521 (263h57m52.756221171s) [noise=true , address=77.90.31.11:4554 , piece_id=OQYJJPQRLA64VFXA5VBWDKJ73Q2YMLTRW3UXJVNBQ3BLCZ55TZ5Q, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024997523 (263h57m52.756265174s) [noise=true , address=45.91.116.6:4973 , piece_id=MP533ZUEZDRXYTQPWKM5URFX6BB3WPGAPFHFMUJQVNBZFPTJHICA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025367439 (263h57m52.756635089s) [noise=true , address=77.90.31.19:4841 , piece_id=MY2CTXR6M4Z7JDC5BE3HENPMUOXXSVX22PNG236QOH5F3552PFCQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025414347 (263h57m52.756681997s) [noise=true , address=104.36.82.2:4729 , piece_id=LY2SWRDWXQHES5FTAF5CP6KCNVCY6FHIBKJNLOCEFYSBNBPIPNIA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025761920 (263h57m52.75702958s) [noise=true , address=104.36.82.7:4909 , piece_id=QB6Z7VE6VFC5L3BNINLJTO53YACDUGNBL6U7WUHKSG5HAFVSXAJQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025888298 (263h57m52.757155948s) [noise=true , address=77.90.31.14:4661 , piece_id=WDR237S6GBHWJGAO2L7M5YASTXX6QOLMIMDZM2D57PQHZA7BYB2Q, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026138559 (263h57m52.757406208s) [noise=true , address=77.90.31.12:4592 , piece_id=LYEZMNLTZVT4UQHZL53CGXZXPHLPJPO34EMLPPRARNVULCOQPXKQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026283681 (263h57m52.75755133s) [noise=true , address=104.36.82.5:4837 , piece_id=AM2AXREOT4ZII77PAWUE76BM6AR4YGI3XF5CGUWRKEE5R5FTDIFA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026491962 (263h57m52.757759613s) [noise=true , address=45.91.116.14:4189 , piece_id=QVER7IKF6T66GHOKDYVUVAYBF4OWORQV3IPKXKTREBYXOLKUI6HA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026640120 (263h57m52.757907771s) [noise=true , address=45.91.116.14:4190 , piece_id=3AZCPPFTZFZ7ANINXPAIR433CKJUXEVDB7FYJIQDLHMYIMRFTGKQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026842701 (263h57m52.758110352s) [noise=true , address=77.90.31.13:4628 , piece_id=B2BESUZXB37NOFSOEUPCUXX3AMEZ44S62HIKN7K4P2FD65CPEDPQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026995178 (263h57m52.758262828s) [noise=true , address=104.36.82.7:4912 , piece_id=ADZS3KGOOF4HPEYRXYTTS7WHDRKRBGRIKUKCORRSAENYPHXMEGHA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.027210582 (263h57m52.758478232s) [noise=true , address=104.36.82.13:4185 , piece_id=G6V7LQWJ3UC4PMMPBZWE67KWQXQENEEO4O4763VEGVGEVIDJ5FEA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.027858310 (263h57m52.75912596s) [noise=true , address=45.91.116.4:4939 , piece_id=BTZOJWID4G2ADAZKPQ6PW7SMV42R2FM756KFMCBSBAMFMJDFULWQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.143746757 (263h57m52.875014407s) [upload=274520]: Next returned nil segment Aug 19 17:24:12.143775502 (263h57m52.875043152s) [upload=274520]: Waiting for error group managing segments... Aug 19 17:24:12.157847553 (263h57m52.889115213s) [noise=true , address=104.36.82.13:4185 , piece_id=G6V7LQWJ3UC4PMMPBZWE67KWQXQENEEO4O4763VEGVGEVIDJ5FEA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.158547338 (263h57m52.889814989s) [noise=true , address=104.36.82.5:4837 , piece_id=AM2AXREOT4ZII77PAWUE76BM6AR4YGI3XF5CGUWRKEE5R5FTDIFA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.158808157 (263h57m52.890075818s) [noise=true , address=104.36.82.2:4729 , piece_id=LY2SWRDWXQHES5FTAF5CP6KCNVCY6FHIBKJNLOCEFYSBNBPIPNIA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.159159038 (263h57m52.890426688s) [noise=true , address=104.36.82.7:4912 , piece_id=ADZS3KGOOF4HPEYRXYTTS7WHDRKRBGRIKUKCORRSAENYPHXMEGHA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.160381044 (263h57m52.891648694s) [noise=true , address=104.36.82.7:4909 , piece_id=QB6Z7VE6VFC5L3BNINLJTO53YACDUGNBL6U7WUHKSG5HAFVSXAJQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.163470309 (263h57m52.894737959s) [noise=true , address=104.36.82.9:4007 , piece_id=YEAQAGKXBODCVNIAG2S7TT5MP3TKTC5PAWGA7CIIBEW5DYS7UQCQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:41.979520414 (263h58m22.710788064s) [noise=true , address=77.90.31.13:4628 , piece_id=B2BESUZXB37NOFSOEUPCUXX3AMEZ44S62HIKN7K4P2FD65CPEDPQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 5.161.89.252:36766->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980102297 (263h58m22.711369958s) [noise=true , address=45.91.116.14:4190 , piece_id=3AZCPPFTZFZ7ANINXPAIR433CKJUXEVDB7FYJIQDLHMYIMRFTGKQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:21646->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980158464 (263h58m22.711426114s) [noise=true , address=45.91.116.4:4939 , piece_id=BTZOJWID4G2ADAZKPQ6PW7SMV42R2FM756KFMCBSBAMFMJDFULWQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 5.161.89.252:51578->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980204240 (263h58m22.71147189s) [noise=true , address=77.90.31.11:4554 , piece_id=OQYJJPQRLA64VFXA5VBWDKJ73Q2YMLTRW3UXJVNBQ3BLCZ55TZ5Q, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 5.161.89.252:9778->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980249345 (263h58m22.711517004s) [noise=true , address=77.90.31.14:4661 , piece_id=WDR237S6GBHWJGAO2L7M5YASTXX6QOLMIMDZM2D57PQHZA7BYB2Q, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:22472->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980292254 (263h58m22.711559905s) [noise=true , address=45.91.116.14:4189 , piece_id=QVER7IKF6T66GHOKDYVUVAYBF4OWORQV3IPKXKTREBYXOLKUI6HA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: read tcp 5.161.89.252:34124->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980365091 (263h58m22.711632752s) [noise=true , address=45.91.116.8:4045 , piece_id=X2HZK7QLI7RRZMF6WYSHK4TYUCPWMF7XRY6KVWKG7SOMGZK4HZXA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: write tcp 5.161.89.252:8238->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980403213 (263h58m22.711670863s) [noise=true , address=77.90.31.19:4841 , piece_id=MY2CTXR6M4Z7JDC5BE3HENPMUOXXSVX22PNG236QOH5F3552PFCQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980547475 (263h58m22.711815124s) [noise=true , address=45.91.116.6:4973 , piece_id=MP533ZUEZDRXYTQPWKM5URFX6BB3WPGAPFHFMUJQVNBZFPTJHICA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 5.161.89.252:29404->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980632083 (263h58m22.711899743s) [noise=true , address=77.90.31.12:4592 , piece_id=LYEZMNLTZVT4UQHZL53CGXZXPHLPJPO34EMLPPRARNVULCOQPXKQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980736129 (263h58m22.712003779s) [seg_pos={0 0}, upload=274520]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:41.997898729 (263h58m22.729166379s) [upload=274520]: Done uploading segments... Aug 19 17:24:41.997919457 (263h58m22.729187108s) [upload=274520]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 389.367546ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................................X...........X..............] duration: 136.708561ms FAIL (2025-08-19 17:29:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:16:50.148435305 (263h50m30.879702956s) [upload=274393]: Starting upload Aug 19 17:16:50.148525875 (263h50m30.879793526s) [upload=274393]: Done starting upload Aug 19 17:16:50.148615374 (263h50m30.879883023s) [upload=274393]: Uploading segments... Aug 19 17:16:50.148667010 (263h50m30.879934661s) [upload=274393]: Got next segment. Inline: true Aug 19 17:16:50.148674014 (263h50m30.879941663s) [upload=274393]: Waiting for error group managing segments... Aug 19 17:16:50.148720332 (263h50m30.879987981s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:16:50.148725801 (263h50m30.879993452s) [upload=274393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:16:50.148729298 (263h50m30.879996948s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:16:50.221372078 (263h50m30.952639728s) [upload=274393]: Done uploading segments... Aug 19 17:16:50.221390462 (263h50m30.952658113s) [upload=274393]: Upload finished. err: metaclient: Too Many Requests Duration:* 136.730101ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 411.791427ms FAIL (2025-08-19 17:29:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:20:30.724709450 (263h54m11.4559771s) [seg_pos={0 0}, upload=274457]: Begin upload segment... Aug 19 17:20:30.724753462 (263h54m11.456021113s) [seg_pos={0 0}, upload=274457]: Done begin upload segment. Aug 19 17:20:30.724781936 (263h54m11.456049586s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.724797054 (263h54m11.456064674s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725166569 (263h54m11.456434189s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725324816 (263h54m11.456592476s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725415496 (263h54m11.456683126s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725643935 (263h54m11.456911565s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725817342 (263h54m11.457084992s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725853730 (263h54m11.45712136s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725891440 (263h54m11.457159071s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726110873 (263h54m11.457378504s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726322932 (263h54m11.457590582s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726338811 (263h54m11.457606432s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726711522 (263h54m11.457979172s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727114529 (263h54m11.458382178s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727561679 (263h54m11.458829329s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728022655 (263h54m11.459290316s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728544566 (263h54m11.459812225s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728932415 (263h54m11.460200435s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.813701018 (263h54m11.544968628s) [upload=274457]: Next returned nil segment Aug 19 17:20:30.813711287 (263h54m11.544978897s) [upload=274457]: Waiting for error group managing segments... Aug 19 17:20:30.826872026 (263h54m11.558139646s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.832633702 (263h54m11.563901332s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.834038051 (263h54m11.565305681s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.844605367 (263h54m11.575872987s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.864222729 (263h54m11.59549035s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.866711214 (263h54m11.597978835s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.882824352 (263h54m11.614091972s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:31.250301081 (263h54m11.981568741s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:11.439696042 (263h54m52.170963692s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:13.585367789 (263h54m54.316635419s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:16.506635739 (263h54m57.237903369s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:22.880177884 (263h55m3.611445533s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:30.678773057 (263h55m11.410040716s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:14fc8QANkMVxxrHcXTWi1yyDhE2gz8BJNwRuvJNfX56TyQCWYZ, address:45.91.116.7:4032): protocol: rpc: dial tcp 45.91.116.7:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678847767 (263h55m11.410115416s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 5.161.89.252:32834->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678926845 (263h55m11.410194495s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: rpc: dial tcp 77.90.31.21:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678951581 (263h55m11.410219241s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, address:77.90.31.18:4829): protocol: rpc: dial tcp 77.90.31.18:4829: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678966439 (263h55m11.410234099s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: write tcp 5.161.89.252:42466->77.90.31.14:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679024038 (263h55m11.410291688s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7, address:45.91.116.10:4107): protocol: read tcp 5.161.89.252:39718->45.91.116.10:4107: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679071326 (263h55m11.410338977s) [seg_pos={0 0}, upload=274457]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 17:21:30.694027780 (263h55m11.425295431s) [upload=274457]: Done uploading segments... Aug 19 17:21:30.694046335 (263h55m11.425313986s) [upload=274457]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 411.810312ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X....................................X.X.X......] duration: 713.252938ms FAIL (2025-08-19 17:28:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 19 17:24:12.613365846 (263h57m53.344633476s) [upload=274521]: Starting upload Aug 19 17:24:12.613406413 (263h57m53.344674043s) [upload=274521]: Done starting upload Aug 19 17:24:12.613493606 (263h57m53.344761226s) [upload=274521]: Uploading segments... Aug 19 17:24:12.613734899 (263h57m53.345002529s) [upload=274521]: Got next segment. Inline: false Aug 19 17:24:12.613764896 (263h57m53.345032526s) [upload=274521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:12.613769284 (263h57m53.345036914s) [upload=274521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:12.651951109 (263h57m53.383218739s) [seg_pos={0 0}, upload=274521]: Begin upload segment... Aug 19 17:24:12.651990573 (263h57m53.383258193s) [seg_pos={0 0}, upload=274521]: Done begin upload segment. Aug 19 17:24:12.652011803 (263h57m53.383279433s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652035357 (263h57m53.383302987s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652327476 (263h57m53.383595106s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652383030 (263h57m53.38365066s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652565463 (263h57m53.383833093s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652676883 (263h57m53.383944513s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652812508 (263h57m53.384080138s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652949715 (263h57m53.384217345s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653091241 (263h57m53.384358871s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653251271 (263h57m53.384518892s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653305264 (263h57m53.384572884s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653509577 (263h57m53.384777207s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653518354 (263h57m53.384785984s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653743006 (263h57m53.385010626s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653788721 (263h57m53.385056351s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653985512 (263h57m53.385253132s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654086841 (263h57m53.385354471s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654373080 (263h57m53.3856407s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.735474331 (263h57m53.466741961s) [upload=274521]: Next returned nil segment Aug 19 17:24:12.735486304 (263h57m53.466753934s) [upload=274521]: Waiting for error group managing segments... Aug 19 17:24:12.756883830 (263h57m53.48815145s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.766603935 (263h57m53.497871585s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.809581250 (263h57m53.540859941s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.908061427 (263h57m53.639329057s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.147463951 (263h57m53.878731601s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.364439955 (263h57m54.095707575s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:51.706547376 (263h58m32.437815006s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:52.370283683 (263h58m33.101551303s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.816229069 (263h58m36.547496699s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.944464809 (263h58m36.675732439s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.083334942 (263h58m48.814602572s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.405488480 (263h58m49.136756101s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.160489383 (263h58m52.891757013s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.595037822 (263h58m53.326305452s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595089570 (263h58m53.326357189s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 5.161.89.252:56044->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595102303 (263h58m53.326369933s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: rpc: dial tcp 45.91.116.4:4950: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595112262 (263h58m53.326379892s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 5.161.89.252:9456->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595145234 (263h58m53.326412864s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1gnJixpV2uZqEMTxi22y1EAzKCHXaQeBWakhT6AcEkrENWT8CT, address:45.91.116.13:4182): protocol: write tcp 5.161.89.252:9502->45.91.116.13:4182: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595163289 (263h58m53.326430909s) [seg_pos={0 0}, upload=274521]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 19 17:25:12.636371896 (263h58m53.367639527s) [upload=274521]: Done uploading segments... Aug 19 17:25:12.636391424 (263h58m53.367659054s) [upload=274521]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 713.274789ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..............................................X....X......] duration: 576.45538ms FAIL (2025-08-19 17:28:46 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== [seg_pos={0 0}, upload=274520]: Done begin upload segment. Aug 19 17:24:12.024531859 (263h57m52.755799509s) [noise=true , address=104.36.82.9:4007 , piece_id=YEAQAGKXBODCVNIAG2S7TT5MP3TKTC5PAWGA7CIIBEW5DYS7UQCQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024547178 (263h57m52.755814828s) [noise=true , address=45.91.116.8:4045 , piece_id=X2HZK7QLI7RRZMF6WYSHK4TYUCPWMF7XRY6KVWKG7SOMGZK4HZXA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024953521 (263h57m52.756221171s) [noise=true , address=77.90.31.11:4554 , piece_id=OQYJJPQRLA64VFXA5VBWDKJ73Q2YMLTRW3UXJVNBQ3BLCZ55TZ5Q, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024997523 (263h57m52.756265174s) [noise=true , address=45.91.116.6:4973 , piece_id=MP533ZUEZDRXYTQPWKM5URFX6BB3WPGAPFHFMUJQVNBZFPTJHICA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025367439 (263h57m52.756635089s) [noise=true , address=77.90.31.19:4841 , piece_id=MY2CTXR6M4Z7JDC5BE3HENPMUOXXSVX22PNG236QOH5F3552PFCQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025414347 (263h57m52.756681997s) [noise=true , address=104.36.82.2:4729 , piece_id=LY2SWRDWXQHES5FTAF5CP6KCNVCY6FHIBKJNLOCEFYSBNBPIPNIA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025761920 (263h57m52.75702958s) [noise=true , address=104.36.82.7:4909 , piece_id=QB6Z7VE6VFC5L3BNINLJTO53YACDUGNBL6U7WUHKSG5HAFVSXAJQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025888298 (263h57m52.757155948s) [noise=true , address=77.90.31.14:4661 , piece_id=WDR237S6GBHWJGAO2L7M5YASTXX6QOLMIMDZM2D57PQHZA7BYB2Q, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026138559 (263h57m52.757406208s) [noise=true , address=77.90.31.12:4592 , piece_id=LYEZMNLTZVT4UQHZL53CGXZXPHLPJPO34EMLPPRARNVULCOQPXKQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026283681 (263h57m52.75755133s) [noise=true , address=104.36.82.5:4837 , piece_id=AM2AXREOT4ZII77PAWUE76BM6AR4YGI3XF5CGUWRKEE5R5FTDIFA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026491962 (263h57m52.757759613s) [noise=true , address=45.91.116.14:4189 , piece_id=QVER7IKF6T66GHOKDYVUVAYBF4OWORQV3IPKXKTREBYXOLKUI6HA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026640120 (263h57m52.757907771s) [noise=true , address=45.91.116.14:4190 , piece_id=3AZCPPFTZFZ7ANINXPAIR433CKJUXEVDB7FYJIQDLHMYIMRFTGKQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026842701 (263h57m52.758110352s) [noise=true , address=77.90.31.13:4628 , piece_id=B2BESUZXB37NOFSOEUPCUXX3AMEZ44S62HIKN7K4P2FD65CPEDPQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026995178 (263h57m52.758262828s) [noise=true , address=104.36.82.7:4912 , piece_id=ADZS3KGOOF4HPEYRXYTTS7WHDRKRBGRIKUKCORRSAENYPHXMEGHA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.027210582 (263h57m52.758478232s) [noise=true , address=104.36.82.13:4185 , piece_id=G6V7LQWJ3UC4PMMPBZWE67KWQXQENEEO4O4763VEGVGEVIDJ5FEA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.027858310 (263h57m52.75912596s) [noise=true , address=45.91.116.4:4939 , piece_id=BTZOJWID4G2ADAZKPQ6PW7SMV42R2FM756KFMCBSBAMFMJDFULWQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.143746757 (263h57m52.875014407s) [upload=274520]: Next returned nil segment Aug 19 17:24:12.143775502 (263h57m52.875043152s) [upload=274520]: Waiting for error group managing segments... Aug 19 17:24:12.157847553 (263h57m52.889115213s) [noise=true , address=104.36.82.13:4185 , piece_id=G6V7LQWJ3UC4PMMPBZWE67KWQXQENEEO4O4763VEGVGEVIDJ5FEA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.158547338 (263h57m52.889814989s) [noise=true , address=104.36.82.5:4837 , piece_id=AM2AXREOT4ZII77PAWUE76BM6AR4YGI3XF5CGUWRKEE5R5FTDIFA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.158808157 (263h57m52.890075818s) [noise=true , address=104.36.82.2:4729 , piece_id=LY2SWRDWXQHES5FTAF5CP6KCNVCY6FHIBKJNLOCEFYSBNBPIPNIA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.159159038 (263h57m52.890426688s) [noise=true , address=104.36.82.7:4912 , piece_id=ADZS3KGOOF4HPEYRXYTTS7WHDRKRBGRIKUKCORRSAENYPHXMEGHA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.160381044 (263h57m52.891648694s) [noise=true , address=104.36.82.7:4909 , piece_id=QB6Z7VE6VFC5L3BNINLJTO53YACDUGNBL6U7WUHKSG5HAFVSXAJQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.163470309 (263h57m52.894737959s) [noise=true , address=104.36.82.9:4007 , piece_id=YEAQAGKXBODCVNIAG2S7TT5MP3TKTC5PAWGA7CIIBEW5DYS7UQCQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:41.979520414 (263h58m22.710788064s) [noise=true , address=77.90.31.13:4628 , piece_id=B2BESUZXB37NOFSOEUPCUXX3AMEZ44S62HIKN7K4P2FD65CPEDPQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 5.161.89.252:36766->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980102297 (263h58m22.711369958s) [noise=true , address=45.91.116.14:4190 , piece_id=3AZCPPFTZFZ7ANINXPAIR433CKJUXEVDB7FYJIQDLHMYIMRFTGKQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:21646->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980158464 (263h58m22.711426114s) [noise=true , address=45.91.116.4:4939 , piece_id=BTZOJWID4G2ADAZKPQ6PW7SMV42R2FM756KFMCBSBAMFMJDFULWQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 5.161.89.252:51578->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980204240 (263h58m22.71147189s) [noise=true , address=77.90.31.11:4554 , piece_id=OQYJJPQRLA64VFXA5VBWDKJ73Q2YMLTRW3UXJVNBQ3BLCZ55TZ5Q, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 5.161.89.252:9778->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980249345 (263h58m22.711517004s) [noise=true , address=77.90.31.14:4661 , piece_id=WDR237S6GBHWJGAO2L7M5YASTXX6QOLMIMDZM2D57PQHZA7BYB2Q, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:22472->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980292254 (263h58m22.711559905s) [noise=true , address=45.91.116.14:4189 , piece_id=QVER7IKF6T66GHOKDYVUVAYBF4OWORQV3IPKXKTREBYXOLKUI6HA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: read tcp 5.161.89.252:34124->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980365091 (263h58m22.711632752s) [noise=true , address=45.91.116.8:4045 , piece_id=X2HZK7QLI7RRZMF6WYSHK4TYUCPWMF7XRY6KVWKG7SOMGZK4HZXA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: write tcp 5.161.89.252:8238->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980403213 (263h58m22.711670863s) [noise=true , address=77.90.31.19:4841 , piece_id=MY2CTXR6M4Z7JDC5BE3HENPMUOXXSVX22PNG236QOH5F3552PFCQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980547475 (263h58m22.711815124s) [noise=true , address=45.91.116.6:4973 , piece_id=MP533ZUEZDRXYTQPWKM5URFX6BB3WPGAPFHFMUJQVNBZFPTJHICA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 5.161.89.252:29404->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980632083 (263h58m22.711899743s) [noise=true , address=77.90.31.12:4592 , piece_id=LYEZMNLTZVT4UQHZL53CGXZXPHLPJPO34EMLPPRARNVULCOQPXKQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980736129 (263h58m22.712003779s) [seg_pos={0 0}, upload=274520]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:41.997898729 (263h58m22.729166379s) [upload=274520]: Done uploading segments... Aug 19 17:24:41.997919457 (263h58m22.729187108s) [upload=274520]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 576.475227ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.579513487s FAIL (2025-08-19 17:28:46 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 1.57955275s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................................X...........X.............] duration: 570.834444ms FAIL (2025-08-19 17:28:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:16:50.148435305 (263h50m30.879702956s) [upload=274393]: Starting upload Aug 19 17:16:50.148525875 (263h50m30.879793526s) [upload=274393]: Done starting upload Aug 19 17:16:50.148615374 (263h50m30.879883023s) [upload=274393]: Uploading segments... Aug 19 17:16:50.148667010 (263h50m30.879934661s) [upload=274393]: Got next segment. Inline: true Aug 19 17:16:50.148674014 (263h50m30.879941663s) [upload=274393]: Waiting for error group managing segments... Aug 19 17:16:50.148720332 (263h50m30.879987981s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:16:50.148725801 (263h50m30.879993452s) [upload=274393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:16:50.148729298 (263h50m30.879996948s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:16:50.221372078 (263h50m30.952639728s) [upload=274393]: Done uploading segments... Aug 19 17:16:50.221390462 (263h50m30.952658113s) [upload=274393]: Upload finished. err: metaclient: Too Many Requests Duration:* 570.854442ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XX..............................................X...X......] duration: 387.358692ms FAIL (2025-08-19 17:28:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 387.375915ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 648.740397ms FAIL (2025-08-19 17:28:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:20:30.724709450 (263h54m11.4559771s) [seg_pos={0 0}, upload=274457]: Begin upload segment... Aug 19 17:20:30.724753462 (263h54m11.456021113s) [seg_pos={0 0}, upload=274457]: Done begin upload segment. Aug 19 17:20:30.724781936 (263h54m11.456049586s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.724797054 (263h54m11.456064674s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725166569 (263h54m11.456434189s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725324816 (263h54m11.456592476s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725415496 (263h54m11.456683126s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725643935 (263h54m11.456911565s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725817342 (263h54m11.457084992s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725853730 (263h54m11.45712136s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725891440 (263h54m11.457159071s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726110873 (263h54m11.457378504s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726322932 (263h54m11.457590582s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726338811 (263h54m11.457606432s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726711522 (263h54m11.457979172s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727114529 (263h54m11.458382178s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727561679 (263h54m11.458829329s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728022655 (263h54m11.459290316s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728544566 (263h54m11.459812225s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728932415 (263h54m11.460200435s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.813701018 (263h54m11.544968628s) [upload=274457]: Next returned nil segment Aug 19 17:20:30.813711287 (263h54m11.544978897s) [upload=274457]: Waiting for error group managing segments... Aug 19 17:20:30.826872026 (263h54m11.558139646s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.832633702 (263h54m11.563901332s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.834038051 (263h54m11.565305681s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.844605367 (263h54m11.575872987s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.864222729 (263h54m11.59549035s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.866711214 (263h54m11.597978835s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.882824352 (263h54m11.614091972s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:31.250301081 (263h54m11.981568741s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:11.439696042 (263h54m52.170963692s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:13.585367789 (263h54m54.316635419s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:16.506635739 (263h54m57.237903369s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:22.880177884 (263h55m3.611445533s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:30.678773057 (263h55m11.410040716s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:14fc8QANkMVxxrHcXTWi1yyDhE2gz8BJNwRuvJNfX56TyQCWYZ, address:45.91.116.7:4032): protocol: rpc: dial tcp 45.91.116.7:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678847767 (263h55m11.410115416s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 5.161.89.252:32834->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678926845 (263h55m11.410194495s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: rpc: dial tcp 77.90.31.21:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678951581 (263h55m11.410219241s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, address:77.90.31.18:4829): protocol: rpc: dial tcp 77.90.31.18:4829: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678966439 (263h55m11.410234099s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: write tcp 5.161.89.252:42466->77.90.31.14:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679024038 (263h55m11.410291688s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7, address:45.91.116.10:4107): protocol: read tcp 5.161.89.252:39718->45.91.116.10:4107: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679071326 (263h55m11.410338977s) [seg_pos={0 0}, upload=274457]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 17:21:30.694027780 (263h55m11.425295431s) [upload=274457]: Done uploading segments... Aug 19 17:21:30.694046335 (263h55m11.425313986s) [upload=274457]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 648.766605ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...........X............] duration: 151.850317ms FAIL (2025-08-19 17:27:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:16:50.148435305 (263h50m30.879702956s) [upload=274393]: Starting upload Aug 19 17:16:50.148525875 (263h50m30.879793526s) [upload=274393]: Done starting upload Aug 19 17:16:50.148615374 (263h50m30.879883023s) [upload=274393]: Uploading segments... Aug 19 17:16:50.148667010 (263h50m30.879934661s) [upload=274393]: Got next segment. Inline: true Aug 19 17:16:50.148674014 (263h50m30.879941663s) [upload=274393]: Waiting for error group managing segments... Aug 19 17:16:50.148720332 (263h50m30.879987981s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:16:50.148725801 (263h50m30.879993452s) [upload=274393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:16:50.148729298 (263h50m30.879996948s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:16:50.221372078 (263h50m30.952639728s) [upload=274393]: Done uploading segments... Aug 19 17:16:50.221390462 (263h50m30.952658113s) [upload=274393]: Upload finished. err: metaclient: Too Many Requests Duration:* 151.87322ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X....................................X.X.X.....] duration: 740.169188ms FAIL (2025-08-19 17:27:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 19 17:24:12.613365846 (263h57m53.344633476s) [upload=274521]: Starting upload Aug 19 17:24:12.613406413 (263h57m53.344674043s) [upload=274521]: Done starting upload Aug 19 17:24:12.613493606 (263h57m53.344761226s) [upload=274521]: Uploading segments... Aug 19 17:24:12.613734899 (263h57m53.345002529s) [upload=274521]: Got next segment. Inline: false Aug 19 17:24:12.613764896 (263h57m53.345032526s) [upload=274521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:12.613769284 (263h57m53.345036914s) [upload=274521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:12.651951109 (263h57m53.383218739s) [seg_pos={0 0}, upload=274521]: Begin upload segment... Aug 19 17:24:12.651990573 (263h57m53.383258193s) [seg_pos={0 0}, upload=274521]: Done begin upload segment. Aug 19 17:24:12.652011803 (263h57m53.383279433s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652035357 (263h57m53.383302987s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652327476 (263h57m53.383595106s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652383030 (263h57m53.38365066s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652565463 (263h57m53.383833093s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652676883 (263h57m53.383944513s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652812508 (263h57m53.384080138s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652949715 (263h57m53.384217345s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653091241 (263h57m53.384358871s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653251271 (263h57m53.384518892s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653305264 (263h57m53.384572884s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653509577 (263h57m53.384777207s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653518354 (263h57m53.384785984s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653743006 (263h57m53.385010626s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653788721 (263h57m53.385056351s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653985512 (263h57m53.385253132s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654086841 (263h57m53.385354471s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654373080 (263h57m53.3856407s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.735474331 (263h57m53.466741961s) [upload=274521]: Next returned nil segment Aug 19 17:24:12.735486304 (263h57m53.466753934s) [upload=274521]: Waiting for error group managing segments... Aug 19 17:24:12.756883830 (263h57m53.48815145s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.766603935 (263h57m53.497871585s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.809581250 (263h57m53.540859941s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.908061427 (263h57m53.639329057s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.147463951 (263h57m53.878731601s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.364439955 (263h57m54.095707575s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:51.706547376 (263h58m32.437815006s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:52.370283683 (263h58m33.101551303s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.816229069 (263h58m36.547496699s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.944464809 (263h58m36.675732439s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.083334942 (263h58m48.814602572s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.405488480 (263h58m49.136756101s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.160489383 (263h58m52.891757013s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.595037822 (263h58m53.326305452s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595089570 (263h58m53.326357189s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 5.161.89.252:56044->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595102303 (263h58m53.326369933s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: rpc: dial tcp 45.91.116.4:4950: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595112262 (263h58m53.326379892s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 5.161.89.252:9456->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595145234 (263h58m53.326412864s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1gnJixpV2uZqEMTxi22y1EAzKCHXaQeBWakhT6AcEkrENWT8CT, address:45.91.116.13:4182): protocol: write tcp 5.161.89.252:9502->45.91.116.13:4182: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595163289 (263h58m53.326430909s) [seg_pos={0 0}, upload=274521]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 19 17:25:12.636371896 (263h58m53.367639527s) [upload=274521]: Done uploading segments... Aug 19 17:25:12.636391424 (263h58m53.367659054s) [upload=274521]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 740.19687ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 1.597666559s FAIL (2025-08-19 17:27:46 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 1.597686386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..............................................X....X.....] duration: 405.795336ms FAIL (2025-08-19 17:27:42 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== [seg_pos={0 0}, upload=274520]: Done begin upload segment. Aug 19 17:24:12.024531859 (263h57m52.755799509s) [noise=true , address=104.36.82.9:4007 , piece_id=YEAQAGKXBODCVNIAG2S7TT5MP3TKTC5PAWGA7CIIBEW5DYS7UQCQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024547178 (263h57m52.755814828s) [noise=true , address=45.91.116.8:4045 , piece_id=X2HZK7QLI7RRZMF6WYSHK4TYUCPWMF7XRY6KVWKG7SOMGZK4HZXA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024953521 (263h57m52.756221171s) [noise=true , address=77.90.31.11:4554 , piece_id=OQYJJPQRLA64VFXA5VBWDKJ73Q2YMLTRW3UXJVNBQ3BLCZ55TZ5Q, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024997523 (263h57m52.756265174s) [noise=true , address=45.91.116.6:4973 , piece_id=MP533ZUEZDRXYTQPWKM5URFX6BB3WPGAPFHFMUJQVNBZFPTJHICA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025367439 (263h57m52.756635089s) [noise=true , address=77.90.31.19:4841 , piece_id=MY2CTXR6M4Z7JDC5BE3HENPMUOXXSVX22PNG236QOH5F3552PFCQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025414347 (263h57m52.756681997s) [noise=true , address=104.36.82.2:4729 , piece_id=LY2SWRDWXQHES5FTAF5CP6KCNVCY6FHIBKJNLOCEFYSBNBPIPNIA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025761920 (263h57m52.75702958s) [noise=true , address=104.36.82.7:4909 , piece_id=QB6Z7VE6VFC5L3BNINLJTO53YACDUGNBL6U7WUHKSG5HAFVSXAJQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025888298 (263h57m52.757155948s) [noise=true , address=77.90.31.14:4661 , piece_id=WDR237S6GBHWJGAO2L7M5YASTXX6QOLMIMDZM2D57PQHZA7BYB2Q, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026138559 (263h57m52.757406208s) [noise=true , address=77.90.31.12:4592 , piece_id=LYEZMNLTZVT4UQHZL53CGXZXPHLPJPO34EMLPPRARNVULCOQPXKQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026283681 (263h57m52.75755133s) [noise=true , address=104.36.82.5:4837 , piece_id=AM2AXREOT4ZII77PAWUE76BM6AR4YGI3XF5CGUWRKEE5R5FTDIFA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026491962 (263h57m52.757759613s) [noise=true , address=45.91.116.14:4189 , piece_id=QVER7IKF6T66GHOKDYVUVAYBF4OWORQV3IPKXKTREBYXOLKUI6HA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026640120 (263h57m52.757907771s) [noise=true , address=45.91.116.14:4190 , piece_id=3AZCPPFTZFZ7ANINXPAIR433CKJUXEVDB7FYJIQDLHMYIMRFTGKQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026842701 (263h57m52.758110352s) [noise=true , address=77.90.31.13:4628 , piece_id=B2BESUZXB37NOFSOEUPCUXX3AMEZ44S62HIKN7K4P2FD65CPEDPQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026995178 (263h57m52.758262828s) [noise=true , address=104.36.82.7:4912 , piece_id=ADZS3KGOOF4HPEYRXYTTS7WHDRKRBGRIKUKCORRSAENYPHXMEGHA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.027210582 (263h57m52.758478232s) [noise=true , address=104.36.82.13:4185 , piece_id=G6V7LQWJ3UC4PMMPBZWE67KWQXQENEEO4O4763VEGVGEVIDJ5FEA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.027858310 (263h57m52.75912596s) [noise=true , address=45.91.116.4:4939 , piece_id=BTZOJWID4G2ADAZKPQ6PW7SMV42R2FM756KFMCBSBAMFMJDFULWQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.143746757 (263h57m52.875014407s) [upload=274520]: Next returned nil segment Aug 19 17:24:12.143775502 (263h57m52.875043152s) [upload=274520]: Waiting for error group managing segments... Aug 19 17:24:12.157847553 (263h57m52.889115213s) [noise=true , address=104.36.82.13:4185 , piece_id=G6V7LQWJ3UC4PMMPBZWE67KWQXQENEEO4O4763VEGVGEVIDJ5FEA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.158547338 (263h57m52.889814989s) [noise=true , address=104.36.82.5:4837 , piece_id=AM2AXREOT4ZII77PAWUE76BM6AR4YGI3XF5CGUWRKEE5R5FTDIFA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.158808157 (263h57m52.890075818s) [noise=true , address=104.36.82.2:4729 , piece_id=LY2SWRDWXQHES5FTAF5CP6KCNVCY6FHIBKJNLOCEFYSBNBPIPNIA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.159159038 (263h57m52.890426688s) [noise=true , address=104.36.82.7:4912 , piece_id=ADZS3KGOOF4HPEYRXYTTS7WHDRKRBGRIKUKCORRSAENYPHXMEGHA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.160381044 (263h57m52.891648694s) [noise=true , address=104.36.82.7:4909 , piece_id=QB6Z7VE6VFC5L3BNINLJTO53YACDUGNBL6U7WUHKSG5HAFVSXAJQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.163470309 (263h57m52.894737959s) [noise=true , address=104.36.82.9:4007 , piece_id=YEAQAGKXBODCVNIAG2S7TT5MP3TKTC5PAWGA7CIIBEW5DYS7UQCQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:41.979520414 (263h58m22.710788064s) [noise=true , address=77.90.31.13:4628 , piece_id=B2BESUZXB37NOFSOEUPCUXX3AMEZ44S62HIKN7K4P2FD65CPEDPQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 5.161.89.252:36766->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980102297 (263h58m22.711369958s) [noise=true , address=45.91.116.14:4190 , piece_id=3AZCPPFTZFZ7ANINXPAIR433CKJUXEVDB7FYJIQDLHMYIMRFTGKQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:21646->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980158464 (263h58m22.711426114s) [noise=true , address=45.91.116.4:4939 , piece_id=BTZOJWID4G2ADAZKPQ6PW7SMV42R2FM756KFMCBSBAMFMJDFULWQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 5.161.89.252:51578->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980204240 (263h58m22.71147189s) [noise=true , address=77.90.31.11:4554 , piece_id=OQYJJPQRLA64VFXA5VBWDKJ73Q2YMLTRW3UXJVNBQ3BLCZ55TZ5Q, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 5.161.89.252:9778->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980249345 (263h58m22.711517004s) [noise=true , address=77.90.31.14:4661 , piece_id=WDR237S6GBHWJGAO2L7M5YASTXX6QOLMIMDZM2D57PQHZA7BYB2Q, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:22472->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980292254 (263h58m22.711559905s) [noise=true , address=45.91.116.14:4189 , piece_id=QVER7IKF6T66GHOKDYVUVAYBF4OWORQV3IPKXKTREBYXOLKUI6HA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: read tcp 5.161.89.252:34124->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980365091 (263h58m22.711632752s) [noise=true , address=45.91.116.8:4045 , piece_id=X2HZK7QLI7RRZMF6WYSHK4TYUCPWMF7XRY6KVWKG7SOMGZK4HZXA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: write tcp 5.161.89.252:8238->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980403213 (263h58m22.711670863s) [noise=true , address=77.90.31.19:4841 , piece_id=MY2CTXR6M4Z7JDC5BE3HENPMUOXXSVX22PNG236QOH5F3552PFCQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980547475 (263h58m22.711815124s) [noise=true , address=45.91.116.6:4973 , piece_id=MP533ZUEZDRXYTQPWKM5URFX6BB3WPGAPFHFMUJQVNBZFPTJHICA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 5.161.89.252:29404->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980632083 (263h58m22.711899743s) [noise=true , address=77.90.31.12:4592 , piece_id=LYEZMNLTZVT4UQHZL53CGXZXPHLPJPO34EMLPPRARNVULCOQPXKQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980736129 (263h58m22.712003779s) [seg_pos={0 0}, upload=274520]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:41.997898729 (263h58m22.729166379s) [upload=274520]: Done uploading segments... Aug 19 17:24:41.997919457 (263h58m22.729187108s) [upload=274520]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 405.848234ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...........X...........] duration: 328.353088ms FAIL (2025-08-19 17:27:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:16:50.148435305 (263h50m30.879702956s) [upload=274393]: Starting upload Aug 19 17:16:50.148525875 (263h50m30.879793526s) [upload=274393]: Done starting upload Aug 19 17:16:50.148615374 (263h50m30.879883023s) [upload=274393]: Uploading segments... Aug 19 17:16:50.148667010 (263h50m30.879934661s) [upload=274393]: Got next segment. Inline: true Aug 19 17:16:50.148674014 (263h50m30.879941663s) [upload=274393]: Waiting for error group managing segments... Aug 19 17:16:50.148720332 (263h50m30.879987981s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:16:50.148725801 (263h50m30.879993452s) [upload=274393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:16:50.148729298 (263h50m30.879996948s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:16:50.221372078 (263h50m30.952639728s) [upload=274393]: Done uploading segments... Aug 19 17:16:50.221390462 (263h50m30.952658113s) [upload=274393]: Upload finished. err: metaclient: Too Many Requests Duration:* 328.396349ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 493.995059ms FAIL (2025-08-19 17:27:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:20:30.724709450 (263h54m11.4559771s) [seg_pos={0 0}, upload=274457]: Begin upload segment... Aug 19 17:20:30.724753462 (263h54m11.456021113s) [seg_pos={0 0}, upload=274457]: Done begin upload segment. Aug 19 17:20:30.724781936 (263h54m11.456049586s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.724797054 (263h54m11.456064674s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725166569 (263h54m11.456434189s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725324816 (263h54m11.456592476s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725415496 (263h54m11.456683126s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725643935 (263h54m11.456911565s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725817342 (263h54m11.457084992s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725853730 (263h54m11.45712136s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725891440 (263h54m11.457159071s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726110873 (263h54m11.457378504s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726322932 (263h54m11.457590582s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726338811 (263h54m11.457606432s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726711522 (263h54m11.457979172s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727114529 (263h54m11.458382178s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727561679 (263h54m11.458829329s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728022655 (263h54m11.459290316s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728544566 (263h54m11.459812225s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728932415 (263h54m11.460200435s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.813701018 (263h54m11.544968628s) [upload=274457]: Next returned nil segment Aug 19 17:20:30.813711287 (263h54m11.544978897s) [upload=274457]: Waiting for error group managing segments... Aug 19 17:20:30.826872026 (263h54m11.558139646s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.832633702 (263h54m11.563901332s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.834038051 (263h54m11.565305681s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.844605367 (263h54m11.575872987s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.864222729 (263h54m11.59549035s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.866711214 (263h54m11.597978835s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.882824352 (263h54m11.614091972s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:31.250301081 (263h54m11.981568741s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:11.439696042 (263h54m52.170963692s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:13.585367789 (263h54m54.316635419s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:16.506635739 (263h54m57.237903369s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:22.880177884 (263h55m3.611445533s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:30.678773057 (263h55m11.410040716s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:14fc8QANkMVxxrHcXTWi1yyDhE2gz8BJNwRuvJNfX56TyQCWYZ, address:45.91.116.7:4032): protocol: rpc: dial tcp 45.91.116.7:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678847767 (263h55m11.410115416s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 5.161.89.252:32834->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678926845 (263h55m11.410194495s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: rpc: dial tcp 77.90.31.21:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678951581 (263h55m11.410219241s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, address:77.90.31.18:4829): protocol: rpc: dial tcp 77.90.31.18:4829: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678966439 (263h55m11.410234099s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: write tcp 5.161.89.252:42466->77.90.31.14:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679024038 (263h55m11.410291688s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7, address:45.91.116.10:4107): protocol: read tcp 5.161.89.252:39718->45.91.116.10:4107: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679071326 (263h55m11.410338977s) [seg_pos={0 0}, upload=274457]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 17:21:30.694027780 (263h55m11.425295431s) [upload=274457]: Done uploading segments... Aug 19 17:21:30.694046335 (263h55m11.425313986s) [upload=274457]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 494.018383ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XX..............................................X...X.....] duration: 386.486721ms FAIL (2025-08-19 17:27:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 386.559548ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.401518901s FAIL (2025-08-19 17:27:24 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 1.401543327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X..............................................X....X....] duration: 381.936822ms FAIL (2025-08-19 17:27:19 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== [seg_pos={0 0}, upload=274520]: Done begin upload segment. Aug 19 17:24:12.024531859 (263h57m52.755799509s) [noise=true , address=104.36.82.9:4007 , piece_id=YEAQAGKXBODCVNIAG2S7TT5MP3TKTC5PAWGA7CIIBEW5DYS7UQCQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024547178 (263h57m52.755814828s) [noise=true , address=45.91.116.8:4045 , piece_id=X2HZK7QLI7RRZMF6WYSHK4TYUCPWMF7XRY6KVWKG7SOMGZK4HZXA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024953521 (263h57m52.756221171s) [noise=true , address=77.90.31.11:4554 , piece_id=OQYJJPQRLA64VFXA5VBWDKJ73Q2YMLTRW3UXJVNBQ3BLCZ55TZ5Q, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024997523 (263h57m52.756265174s) [noise=true , address=45.91.116.6:4973 , piece_id=MP533ZUEZDRXYTQPWKM5URFX6BB3WPGAPFHFMUJQVNBZFPTJHICA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025367439 (263h57m52.756635089s) [noise=true , address=77.90.31.19:4841 , piece_id=MY2CTXR6M4Z7JDC5BE3HENPMUOXXSVX22PNG236QOH5F3552PFCQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025414347 (263h57m52.756681997s) [noise=true , address=104.36.82.2:4729 , piece_id=LY2SWRDWXQHES5FTAF5CP6KCNVCY6FHIBKJNLOCEFYSBNBPIPNIA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025761920 (263h57m52.75702958s) [noise=true , address=104.36.82.7:4909 , piece_id=QB6Z7VE6VFC5L3BNINLJTO53YACDUGNBL6U7WUHKSG5HAFVSXAJQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025888298 (263h57m52.757155948s) [noise=true , address=77.90.31.14:4661 , piece_id=WDR237S6GBHWJGAO2L7M5YASTXX6QOLMIMDZM2D57PQHZA7BYB2Q, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026138559 (263h57m52.757406208s) [noise=true , address=77.90.31.12:4592 , piece_id=LYEZMNLTZVT4UQHZL53CGXZXPHLPJPO34EMLPPRARNVULCOQPXKQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026283681 (263h57m52.75755133s) [noise=true , address=104.36.82.5:4837 , piece_id=AM2AXREOT4ZII77PAWUE76BM6AR4YGI3XF5CGUWRKEE5R5FTDIFA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026491962 (263h57m52.757759613s) [noise=true , address=45.91.116.14:4189 , piece_id=QVER7IKF6T66GHOKDYVUVAYBF4OWORQV3IPKXKTREBYXOLKUI6HA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026640120 (263h57m52.757907771s) [noise=true , address=45.91.116.14:4190 , piece_id=3AZCPPFTZFZ7ANINXPAIR433CKJUXEVDB7FYJIQDLHMYIMRFTGKQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026842701 (263h57m52.758110352s) [noise=true , address=77.90.31.13:4628 , piece_id=B2BESUZXB37NOFSOEUPCUXX3AMEZ44S62HIKN7K4P2FD65CPEDPQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026995178 (263h57m52.758262828s) [noise=true , address=104.36.82.7:4912 , piece_id=ADZS3KGOOF4HPEYRXYTTS7WHDRKRBGRIKUKCORRSAENYPHXMEGHA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.027210582 (263h57m52.758478232s) [noise=true , address=104.36.82.13:4185 , piece_id=G6V7LQWJ3UC4PMMPBZWE67KWQXQENEEO4O4763VEGVGEVIDJ5FEA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.027858310 (263h57m52.75912596s) [noise=true , address=45.91.116.4:4939 , piece_id=BTZOJWID4G2ADAZKPQ6PW7SMV42R2FM756KFMCBSBAMFMJDFULWQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.143746757 (263h57m52.875014407s) [upload=274520]: Next returned nil segment Aug 19 17:24:12.143775502 (263h57m52.875043152s) [upload=274520]: Waiting for error group managing segments... Aug 19 17:24:12.157847553 (263h57m52.889115213s) [noise=true , address=104.36.82.13:4185 , piece_id=G6V7LQWJ3UC4PMMPBZWE67KWQXQENEEO4O4763VEGVGEVIDJ5FEA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.158547338 (263h57m52.889814989s) [noise=true , address=104.36.82.5:4837 , piece_id=AM2AXREOT4ZII77PAWUE76BM6AR4YGI3XF5CGUWRKEE5R5FTDIFA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.158808157 (263h57m52.890075818s) [noise=true , address=104.36.82.2:4729 , piece_id=LY2SWRDWXQHES5FTAF5CP6KCNVCY6FHIBKJNLOCEFYSBNBPIPNIA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.159159038 (263h57m52.890426688s) [noise=true , address=104.36.82.7:4912 , piece_id=ADZS3KGOOF4HPEYRXYTTS7WHDRKRBGRIKUKCORRSAENYPHXMEGHA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.160381044 (263h57m52.891648694s) [noise=true , address=104.36.82.7:4909 , piece_id=QB6Z7VE6VFC5L3BNINLJTO53YACDUGNBL6U7WUHKSG5HAFVSXAJQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.163470309 (263h57m52.894737959s) [noise=true , address=104.36.82.9:4007 , piece_id=YEAQAGKXBODCVNIAG2S7TT5MP3TKTC5PAWGA7CIIBEW5DYS7UQCQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:41.979520414 (263h58m22.710788064s) [noise=true , address=77.90.31.13:4628 , piece_id=B2BESUZXB37NOFSOEUPCUXX3AMEZ44S62HIKN7K4P2FD65CPEDPQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 5.161.89.252:36766->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980102297 (263h58m22.711369958s) [noise=true , address=45.91.116.14:4190 , piece_id=3AZCPPFTZFZ7ANINXPAIR433CKJUXEVDB7FYJIQDLHMYIMRFTGKQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:21646->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980158464 (263h58m22.711426114s) [noise=true , address=45.91.116.4:4939 , piece_id=BTZOJWID4G2ADAZKPQ6PW7SMV42R2FM756KFMCBSBAMFMJDFULWQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 5.161.89.252:51578->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980204240 (263h58m22.71147189s) [noise=true , address=77.90.31.11:4554 , piece_id=OQYJJPQRLA64VFXA5VBWDKJ73Q2YMLTRW3UXJVNBQ3BLCZ55TZ5Q, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 5.161.89.252:9778->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980249345 (263h58m22.711517004s) [noise=true , address=77.90.31.14:4661 , piece_id=WDR237S6GBHWJGAO2L7M5YASTXX6QOLMIMDZM2D57PQHZA7BYB2Q, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:22472->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980292254 (263h58m22.711559905s) [noise=true , address=45.91.116.14:4189 , piece_id=QVER7IKF6T66GHOKDYVUVAYBF4OWORQV3IPKXKTREBYXOLKUI6HA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: read tcp 5.161.89.252:34124->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980365091 (263h58m22.711632752s) [noise=true , address=45.91.116.8:4045 , piece_id=X2HZK7QLI7RRZMF6WYSHK4TYUCPWMF7XRY6KVWKG7SOMGZK4HZXA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: write tcp 5.161.89.252:8238->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980403213 (263h58m22.711670863s) [noise=true , address=77.90.31.19:4841 , piece_id=MY2CTXR6M4Z7JDC5BE3HENPMUOXXSVX22PNG236QOH5F3552PFCQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980547475 (263h58m22.711815124s) [noise=true , address=45.91.116.6:4973 , piece_id=MP533ZUEZDRXYTQPWKM5URFX6BB3WPGAPFHFMUJQVNBZFPTJHICA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 5.161.89.252:29404->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980632083 (263h58m22.711899743s) [noise=true , address=77.90.31.12:4592 , piece_id=LYEZMNLTZVT4UQHZL53CGXZXPHLPJPO34EMLPPRARNVULCOQPXKQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980736129 (263h58m22.712003779s) [seg_pos={0 0}, upload=274520]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:41.997898729 (263h58m22.729166379s) [upload=274520]: Done uploading segments... Aug 19 17:24:41.997919457 (263h58m22.729187108s) [upload=274520]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 381.958102ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...........X..........] duration: 160.600759ms FAIL (2025-08-19 17:27:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:16:50.148435305 (263h50m30.879702956s) [upload=274393]: Starting upload Aug 19 17:16:50.148525875 (263h50m30.879793526s) [upload=274393]: Done starting upload Aug 19 17:16:50.148615374 (263h50m30.879883023s) [upload=274393]: Uploading segments... Aug 19 17:16:50.148667010 (263h50m30.879934661s) [upload=274393]: Got next segment. Inline: true Aug 19 17:16:50.148674014 (263h50m30.879941663s) [upload=274393]: Waiting for error group managing segments... Aug 19 17:16:50.148720332 (263h50m30.879987981s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:16:50.148725801 (263h50m30.879993452s) [upload=274393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:16:50.148729298 (263h50m30.879996948s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:16:50.221372078 (263h50m30.952639728s) [upload=274393]: Done uploading segments... Aug 19 17:16:50.221390462 (263h50m30.952658113s) [upload=274393]: Upload finished. err: metaclient: Too Many Requests Duration:* 160.627541ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X....................................X.X.X....] duration: 541.469736ms FAIL (2025-08-19 17:27:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 19 17:24:12.613365846 (263h57m53.344633476s) [upload=274521]: Starting upload Aug 19 17:24:12.613406413 (263h57m53.344674043s) [upload=274521]: Done starting upload Aug 19 17:24:12.613493606 (263h57m53.344761226s) [upload=274521]: Uploading segments... Aug 19 17:24:12.613734899 (263h57m53.345002529s) [upload=274521]: Got next segment. Inline: false Aug 19 17:24:12.613764896 (263h57m53.345032526s) [upload=274521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:12.613769284 (263h57m53.345036914s) [upload=274521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:12.651951109 (263h57m53.383218739s) [seg_pos={0 0}, upload=274521]: Begin upload segment... Aug 19 17:24:12.651990573 (263h57m53.383258193s) [seg_pos={0 0}, upload=274521]: Done begin upload segment. Aug 19 17:24:12.652011803 (263h57m53.383279433s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652035357 (263h57m53.383302987s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652327476 (263h57m53.383595106s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652383030 (263h57m53.38365066s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652565463 (263h57m53.383833093s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652676883 (263h57m53.383944513s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652812508 (263h57m53.384080138s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652949715 (263h57m53.384217345s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653091241 (263h57m53.384358871s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653251271 (263h57m53.384518892s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653305264 (263h57m53.384572884s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653509577 (263h57m53.384777207s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653518354 (263h57m53.384785984s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653743006 (263h57m53.385010626s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653788721 (263h57m53.385056351s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653985512 (263h57m53.385253132s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654086841 (263h57m53.385354471s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654373080 (263h57m53.3856407s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.735474331 (263h57m53.466741961s) [upload=274521]: Next returned nil segment Aug 19 17:24:12.735486304 (263h57m53.466753934s) [upload=274521]: Waiting for error group managing segments... Aug 19 17:24:12.756883830 (263h57m53.48815145s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.766603935 (263h57m53.497871585s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.809581250 (263h57m53.540859941s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.908061427 (263h57m53.639329057s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.147463951 (263h57m53.878731601s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.364439955 (263h57m54.095707575s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:51.706547376 (263h58m32.437815006s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:52.370283683 (263h58m33.101551303s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.816229069 (263h58m36.547496699s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.944464809 (263h58m36.675732439s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.083334942 (263h58m48.814602572s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.405488480 (263h58m49.136756101s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.160489383 (263h58m52.891757013s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.595037822 (263h58m53.326305452s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595089570 (263h58m53.326357189s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 5.161.89.252:56044->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595102303 (263h58m53.326369933s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: rpc: dial tcp 45.91.116.4:4950: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595112262 (263h58m53.326379892s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 5.161.89.252:9456->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595145234 (263h58m53.326412864s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1gnJixpV2uZqEMTxi22y1EAzKCHXaQeBWakhT6AcEkrENWT8CT, address:45.91.116.13:4182): protocol: write tcp 5.161.89.252:9502->45.91.116.13:4182: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595163289 (263h58m53.326430909s) [seg_pos={0 0}, upload=274521]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 19 17:25:12.636371896 (263h58m53.367639527s) [upload=274521]: Done uploading segments... Aug 19 17:25:12.636391424 (263h58m53.367659054s) [upload=274521]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 541.494632ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XX..............................................X...X....] duration: 637.052458ms FAIL (2025-08-19 17:27:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 637.072866ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 675.46692ms FAIL (2025-08-19 17:26:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:20:30.724709450 (263h54m11.4559771s) [seg_pos={0 0}, upload=274457]: Begin upload segment... Aug 19 17:20:30.724753462 (263h54m11.456021113s) [seg_pos={0 0}, upload=274457]: Done begin upload segment. Aug 19 17:20:30.724781936 (263h54m11.456049586s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.724797054 (263h54m11.456064674s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725166569 (263h54m11.456434189s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725324816 (263h54m11.456592476s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725415496 (263h54m11.456683126s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725643935 (263h54m11.456911565s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725817342 (263h54m11.457084992s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725853730 (263h54m11.45712136s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725891440 (263h54m11.457159071s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726110873 (263h54m11.457378504s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726322932 (263h54m11.457590582s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726338811 (263h54m11.457606432s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726711522 (263h54m11.457979172s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727114529 (263h54m11.458382178s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727561679 (263h54m11.458829329s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728022655 (263h54m11.459290316s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728544566 (263h54m11.459812225s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728932415 (263h54m11.460200435s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.813701018 (263h54m11.544968628s) [upload=274457]: Next returned nil segment Aug 19 17:20:30.813711287 (263h54m11.544978897s) [upload=274457]: Waiting for error group managing segments... Aug 19 17:20:30.826872026 (263h54m11.558139646s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.832633702 (263h54m11.563901332s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.834038051 (263h54m11.565305681s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.844605367 (263h54m11.575872987s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.864222729 (263h54m11.59549035s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.866711214 (263h54m11.597978835s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.882824352 (263h54m11.614091972s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:31.250301081 (263h54m11.981568741s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:11.439696042 (263h54m52.170963692s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:13.585367789 (263h54m54.316635419s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:16.506635739 (263h54m57.237903369s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:22.880177884 (263h55m3.611445533s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:30.678773057 (263h55m11.410040716s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:14fc8QANkMVxxrHcXTWi1yyDhE2gz8BJNwRuvJNfX56TyQCWYZ, address:45.91.116.7:4032): protocol: rpc: dial tcp 45.91.116.7:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678847767 (263h55m11.410115416s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 5.161.89.252:32834->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678926845 (263h55m11.410194495s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: rpc: dial tcp 77.90.31.21:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678951581 (263h55m11.410219241s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, address:77.90.31.18:4829): protocol: rpc: dial tcp 77.90.31.18:4829: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678966439 (263h55m11.410234099s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: write tcp 5.161.89.252:42466->77.90.31.14:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679024038 (263h55m11.410291688s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7, address:45.91.116.10:4107): protocol: read tcp 5.161.89.252:39718->45.91.116.10:4107: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679071326 (263h55m11.410338977s) [seg_pos={0 0}, upload=274457]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 17:21:30.694027780 (263h55m11.425295431s) [upload=274457]: Done uploading segments... Aug 19 17:21:30.694046335 (263h55m11.425313986s) [upload=274457]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 675.496306ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.701266768s FAIL (2025-08-19 17:26:52 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 1.701311442s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....XX..............................................X...X...] duration: 344.843396ms FAIL (2025-08-19 17:26:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 344.863735ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X....................................X.X.X...] duration: 509.925137ms FAIL (2025-08-19 17:26:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 19 17:24:12.613365846 (263h57m53.344633476s) [upload=274521]: Starting upload Aug 19 17:24:12.613406413 (263h57m53.344674043s) [upload=274521]: Done starting upload Aug 19 17:24:12.613493606 (263h57m53.344761226s) [upload=274521]: Uploading segments... Aug 19 17:24:12.613734899 (263h57m53.345002529s) [upload=274521]: Got next segment. Inline: false Aug 19 17:24:12.613764896 (263h57m53.345032526s) [upload=274521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:12.613769284 (263h57m53.345036914s) [upload=274521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:12.651951109 (263h57m53.383218739s) [seg_pos={0 0}, upload=274521]: Begin upload segment... Aug 19 17:24:12.651990573 (263h57m53.383258193s) [seg_pos={0 0}, upload=274521]: Done begin upload segment. Aug 19 17:24:12.652011803 (263h57m53.383279433s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652035357 (263h57m53.383302987s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652327476 (263h57m53.383595106s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652383030 (263h57m53.38365066s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652565463 (263h57m53.383833093s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652676883 (263h57m53.383944513s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652812508 (263h57m53.384080138s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652949715 (263h57m53.384217345s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653091241 (263h57m53.384358871s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653251271 (263h57m53.384518892s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653305264 (263h57m53.384572884s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653509577 (263h57m53.384777207s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653518354 (263h57m53.384785984s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653743006 (263h57m53.385010626s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653788721 (263h57m53.385056351s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653985512 (263h57m53.385253132s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654086841 (263h57m53.385354471s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654373080 (263h57m53.3856407s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.735474331 (263h57m53.466741961s) [upload=274521]: Next returned nil segment Aug 19 17:24:12.735486304 (263h57m53.466753934s) [upload=274521]: Waiting for error group managing segments... Aug 19 17:24:12.756883830 (263h57m53.48815145s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.766603935 (263h57m53.497871585s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.809581250 (263h57m53.540859941s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.908061427 (263h57m53.639329057s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.147463951 (263h57m53.878731601s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.364439955 (263h57m54.095707575s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:51.706547376 (263h58m32.437815006s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:52.370283683 (263h58m33.101551303s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.816229069 (263h58m36.547496699s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.944464809 (263h58m36.675732439s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.083334942 (263h58m48.814602572s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.405488480 (263h58m49.136756101s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.160489383 (263h58m52.891757013s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.595037822 (263h58m53.326305452s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595089570 (263h58m53.326357189s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 5.161.89.252:56044->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595102303 (263h58m53.326369933s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: rpc: dial tcp 45.91.116.4:4950: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595112262 (263h58m53.326379892s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 5.161.89.252:9456->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595145234 (263h58m53.326412864s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1gnJixpV2uZqEMTxi22y1EAzKCHXaQeBWakhT6AcEkrENWT8CT, address:45.91.116.13:4182): protocol: write tcp 5.161.89.252:9502->45.91.116.13:4182: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595163289 (263h58m53.326430909s) [seg_pos={0 0}, upload=274521]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 19 17:25:12.636371896 (263h58m53.367639527s) [upload=274521]: Done uploading segments... Aug 19 17:25:12.636391424 (263h58m53.367659054s) [upload=274521]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 509.951688ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 427.083071ms FAIL (2025-08-19 17:26:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:20:30.724709450 (263h54m11.4559771s) [seg_pos={0 0}, upload=274457]: Begin upload segment... Aug 19 17:20:30.724753462 (263h54m11.456021113s) [seg_pos={0 0}, upload=274457]: Done begin upload segment. Aug 19 17:20:30.724781936 (263h54m11.456049586s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.724797054 (263h54m11.456064674s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725166569 (263h54m11.456434189s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725324816 (263h54m11.456592476s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725415496 (263h54m11.456683126s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725643935 (263h54m11.456911565s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725817342 (263h54m11.457084992s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725853730 (263h54m11.45712136s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.725891440 (263h54m11.457159071s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726110873 (263h54m11.457378504s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726322932 (263h54m11.457590582s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726338811 (263h54m11.457606432s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.726711522 (263h54m11.457979172s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727114529 (263h54m11.458382178s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.727561679 (263h54m11.458829329s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728022655 (263h54m11.459290316s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728544566 (263h54m11.459812225s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.728932415 (263h54m11.460200435s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Uploading piece... Aug 19 17:20:30.813701018 (263h54m11.544968628s) [upload=274457]: Next returned nil segment Aug 19 17:20:30.813711287 (263h54m11.544978897s) [upload=274457]: Waiting for error group managing segments... Aug 19 17:20:30.826872026 (263h54m11.558139646s) [noise=true , address=104.36.82.23:4458 , piece_id=B6SB2PNKJIT5GLPS7Z2ZFPT7A2PXDFHHC3QHCH2MQ3I4WK7MPRSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.832633702 (263h54m11.563901332s) [noise=true , address=104.36.82.44:4882 , piece_id=FADDUTQAO3INVE2JAWVR4SQISS7DZPV7ROI2FHIDZED6VQCLPKKA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.834038051 (263h54m11.565305681s) [noise=true , address=79.127.219.47:4062 , piece_id=A6756Y5PYSEIPMBUN6SAQ7DXQNADFJS6ECMYKBKSH2MPK6MYUT6A, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.844605367 (263h54m11.575872987s) [noise=true , address=104.36.82.2:4764 , piece_id=CBWOVFNOYNAY5F462E5J4KNX5LLYKZQEEJUOLKXLPDCWWHOMWW2Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.864222729 (263h54m11.59549035s) [noise=true , address=104.36.82.17:4353 , piece_id=X4HQY47TKKQ72QPDBMYFK3JHFKPQVTF5BFSXWYNYGAELKBV75KOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.866711214 (263h54m11.597978835s) [noise=true , address=104.36.82.34:4559 , piece_id=J5RYVJTRFQG6REZAZJW7URTZQHMQON2SUP5QHDIK53VHV3KJCLSA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:30.882824352 (263h54m11.614091972s) [noise=true , address=104.36.82.43:4839 , piece_id=SNPCZ3JHY3V2TNJI2INTRZSBDVXVTFQHKOITBEVAAZ5PFOHPSZSQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:20:31.250301081 (263h54m11.981568741s) [noise=true , address=79.127.205.235:4692 , piece_id=OJJ2PG7H3IBPMW5PFWO4YLQ3KT2D5YFH2PSWCOX3CGUZRFFZWB4Q, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:11.439696042 (263h54m52.170963692s) [noise=true , address=77.90.31.17:4797 , piece_id=AVUG5MVC6F7JVM2AWXCWFG5WZ7XGYT24H5QY36W7DCQQAMSHUTLA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:13.585367789 (263h54m54.316635419s) [noise=true , address=45.91.116.11:4147 , piece_id=23JASHPHO3VFWMEKGIT63COZGTWK3I3EEEKGRS7GX7SIFKIURILQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:16.506635739 (263h54m57.237903369s) [noise=true , address=77.90.31.16:4763 , piece_id=2V5IWUJHLF3YH3ZBQJFNH2JCGZ54SQASHQ5FLLGHEQOOSISFKWJQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:22.880177884 (263h55m3.611445533s) [noise=true , address=77.90.31.17:4783 , piece_id=RYTX2QLEKTYQTOR3NSZIPY2TWTBL2ZUKSJV5ORBKH5R2ZPXUKBNQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: <nil> Aug 19 17:21:30.678773057 (263h55m11.410040716s) [noise=true , address=45.91.116.7:4032 , piece_id=M2BXIGAGRRSUZ7EGWFA2IAFQV7ZRUZCH5NBWS4PYFACOGWN6COZQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:14fc8QANkMVxxrHcXTWi1yyDhE2gz8BJNwRuvJNfX56TyQCWYZ, address:45.91.116.7:4032): protocol: rpc: dial tcp 45.91.116.7:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678847767 (263h55m11.410115416s) [noise=true , address=45.91.116.12:4238 , piece_id=6NKCMPVSRFRDDX3TU3ET2KOLQ2DCU2OVNZI5U6ZLUWP6JTF3PCWQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 5.161.89.252:32834->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678926845 (263h55m11.410194495s) [noise=true , address=77.90.31.21:4890 , piece_id=5CIXXVAF25OX7BV7JI6QQ3UZ6JE62XC2GBZOYUMOQQXT3HUD5JOQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: rpc: dial tcp 77.90.31.21:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678951581 (263h55m11.410219241s) [noise=true , address=77.90.31.18:4829 , piece_id=Y43CQMZRAZPDXAXNJJHFDX3C3LEKTFBBLKNRGMVWM7WUM34TVLHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, address:77.90.31.18:4829): protocol: rpc: dial tcp 77.90.31.18:4829: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.678966439 (263h55m11.410234099s) [noise=true , address=77.90.31.14:4695 , piece_id=I3HQFGDDWHDKIGJELHQXZ6MZQYCZ5TUDG4RK4M2OLJBNAS7TNWHQ, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: write tcp 5.161.89.252:42466->77.90.31.14:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679024038 (263h55m11.410291688s) [noise=true , address=45.91.116.10:4107 , piece_id=5Q5KSIA3G2LXQGXCAJRW2S7SKLOIQQYZ4EHMB3WZEB7UHA3F25WA, seg_pos={0 0}, upload=274457]: Done uploading piece. err: ecclient: upload failed (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7, address:45.91.116.10:4107): protocol: read tcp 5.161.89.252:39718->45.91.116.10:4107: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:30.679071326 (263h55m11.410338977s) [seg_pos={0 0}, upload=274457]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 17:21:30.694027780 (263h55m11.425295431s) [upload=274457]: Done uploading segments... Aug 19 17:21:30.694046335 (263h55m11.425313986s) [upload=274457]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 427.101587ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...........X.........] duration: 251.026036ms FAIL (2025-08-19 17:26:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:16:50.148435305 (263h50m30.879702956s) [upload=274393]: Starting upload Aug 19 17:16:50.148525875 (263h50m30.879793526s) [upload=274393]: Done starting upload Aug 19 17:16:50.148615374 (263h50m30.879883023s) [upload=274393]: Uploading segments... Aug 19 17:16:50.148667010 (263h50m30.879934661s) [upload=274393]: Got next segment. Inline: true Aug 19 17:16:50.148674014 (263h50m30.879941663s) [upload=274393]: Waiting for error group managing segments... Aug 19 17:16:50.148720332 (263h50m30.879987981s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:16:50.148725801 (263h50m30.879993452s) [upload=274393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:16:50.148729298 (263h50m30.879996948s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:16:50.221372078 (263h50m30.952639728s) [upload=274393]: Done uploading segments... Aug 19 17:16:50.221390462 (263h50m30.952658113s) [upload=274393]: Upload finished. err: metaclient: Too Many Requests Duration:* 251.053638ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X..............................................X....X...] duration: 539.705699ms FAIL (2025-08-19 17:26:11 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== [seg_pos={0 0}, upload=274520]: Done begin upload segment. Aug 19 17:24:12.024531859 (263h57m52.755799509s) [noise=true , address=104.36.82.9:4007 , piece_id=YEAQAGKXBODCVNIAG2S7TT5MP3TKTC5PAWGA7CIIBEW5DYS7UQCQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024547178 (263h57m52.755814828s) [noise=true , address=45.91.116.8:4045 , piece_id=X2HZK7QLI7RRZMF6WYSHK4TYUCPWMF7XRY6KVWKG7SOMGZK4HZXA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024953521 (263h57m52.756221171s) [noise=true , address=77.90.31.11:4554 , piece_id=OQYJJPQRLA64VFXA5VBWDKJ73Q2YMLTRW3UXJVNBQ3BLCZ55TZ5Q, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024997523 (263h57m52.756265174s) [noise=true , address=45.91.116.6:4973 , piece_id=MP533ZUEZDRXYTQPWKM5URFX6BB3WPGAPFHFMUJQVNBZFPTJHICA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025367439 (263h57m52.756635089s) [noise=true , address=77.90.31.19:4841 , piece_id=MY2CTXR6M4Z7JDC5BE3HENPMUOXXSVX22PNG236QOH5F3552PFCQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025414347 (263h57m52.756681997s) [noise=true , address=104.36.82.2:4729 , piece_id=LY2SWRDWXQHES5FTAF5CP6KCNVCY6FHIBKJNLOCEFYSBNBPIPNIA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025761920 (263h57m52.75702958s) [noise=true , address=104.36.82.7:4909 , piece_id=QB6Z7VE6VFC5L3BNINLJTO53YACDUGNBL6U7WUHKSG5HAFVSXAJQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025888298 (263h57m52.757155948s) [noise=true , address=77.90.31.14:4661 , piece_id=WDR237S6GBHWJGAO2L7M5YASTXX6QOLMIMDZM2D57PQHZA7BYB2Q, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026138559 (263h57m52.757406208s) [noise=true , address=77.90.31.12:4592 , piece_id=LYEZMNLTZVT4UQHZL53CGXZXPHLPJPO34EMLPPRARNVULCOQPXKQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026283681 (263h57m52.75755133s) [noise=true , address=104.36.82.5:4837 , piece_id=AM2AXREOT4ZII77PAWUE76BM6AR4YGI3XF5CGUWRKEE5R5FTDIFA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026491962 (263h57m52.757759613s) [noise=true , address=45.91.116.14:4189 , piece_id=QVER7IKF6T66GHOKDYVUVAYBF4OWORQV3IPKXKTREBYXOLKUI6HA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026640120 (263h57m52.757907771s) [noise=true , address=45.91.116.14:4190 , piece_id=3AZCPPFTZFZ7ANINXPAIR433CKJUXEVDB7FYJIQDLHMYIMRFTGKQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026842701 (263h57m52.758110352s) [noise=true , address=77.90.31.13:4628 , piece_id=B2BESUZXB37NOFSOEUPCUXX3AMEZ44S62HIKN7K4P2FD65CPEDPQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026995178 (263h57m52.758262828s) [noise=true , address=104.36.82.7:4912 , piece_id=ADZS3KGOOF4HPEYRXYTTS7WHDRKRBGRIKUKCORRSAENYPHXMEGHA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.027210582 (263h57m52.758478232s) [noise=true , address=104.36.82.13:4185 , piece_id=G6V7LQWJ3UC4PMMPBZWE67KWQXQENEEO4O4763VEGVGEVIDJ5FEA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.027858310 (263h57m52.75912596s) [noise=true , address=45.91.116.4:4939 , piece_id=BTZOJWID4G2ADAZKPQ6PW7SMV42R2FM756KFMCBSBAMFMJDFULWQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.143746757 (263h57m52.875014407s) [upload=274520]: Next returned nil segment Aug 19 17:24:12.143775502 (263h57m52.875043152s) [upload=274520]: Waiting for error group managing segments... Aug 19 17:24:12.157847553 (263h57m52.889115213s) [noise=true , address=104.36.82.13:4185 , piece_id=G6V7LQWJ3UC4PMMPBZWE67KWQXQENEEO4O4763VEGVGEVIDJ5FEA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.158547338 (263h57m52.889814989s) [noise=true , address=104.36.82.5:4837 , piece_id=AM2AXREOT4ZII77PAWUE76BM6AR4YGI3XF5CGUWRKEE5R5FTDIFA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.158808157 (263h57m52.890075818s) [noise=true , address=104.36.82.2:4729 , piece_id=LY2SWRDWXQHES5FTAF5CP6KCNVCY6FHIBKJNLOCEFYSBNBPIPNIA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.159159038 (263h57m52.890426688s) [noise=true , address=104.36.82.7:4912 , piece_id=ADZS3KGOOF4HPEYRXYTTS7WHDRKRBGRIKUKCORRSAENYPHXMEGHA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.160381044 (263h57m52.891648694s) [noise=true , address=104.36.82.7:4909 , piece_id=QB6Z7VE6VFC5L3BNINLJTO53YACDUGNBL6U7WUHKSG5HAFVSXAJQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.163470309 (263h57m52.894737959s) [noise=true , address=104.36.82.9:4007 , piece_id=YEAQAGKXBODCVNIAG2S7TT5MP3TKTC5PAWGA7CIIBEW5DYS7UQCQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:41.979520414 (263h58m22.710788064s) [noise=true , address=77.90.31.13:4628 , piece_id=B2BESUZXB37NOFSOEUPCUXX3AMEZ44S62HIKN7K4P2FD65CPEDPQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 5.161.89.252:36766->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980102297 (263h58m22.711369958s) [noise=true , address=45.91.116.14:4190 , piece_id=3AZCPPFTZFZ7ANINXPAIR433CKJUXEVDB7FYJIQDLHMYIMRFTGKQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:21646->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980158464 (263h58m22.711426114s) [noise=true , address=45.91.116.4:4939 , piece_id=BTZOJWID4G2ADAZKPQ6PW7SMV42R2FM756KFMCBSBAMFMJDFULWQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 5.161.89.252:51578->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980204240 (263h58m22.71147189s) [noise=true , address=77.90.31.11:4554 , piece_id=OQYJJPQRLA64VFXA5VBWDKJ73Q2YMLTRW3UXJVNBQ3BLCZ55TZ5Q, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 5.161.89.252:9778->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980249345 (263h58m22.711517004s) [noise=true , address=77.90.31.14:4661 , piece_id=WDR237S6GBHWJGAO2L7M5YASTXX6QOLMIMDZM2D57PQHZA7BYB2Q, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:22472->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980292254 (263h58m22.711559905s) [noise=true , address=45.91.116.14:4189 , piece_id=QVER7IKF6T66GHOKDYVUVAYBF4OWORQV3IPKXKTREBYXOLKUI6HA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: read tcp 5.161.89.252:34124->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980365091 (263h58m22.711632752s) [noise=true , address=45.91.116.8:4045 , piece_id=X2HZK7QLI7RRZMF6WYSHK4TYUCPWMF7XRY6KVWKG7SOMGZK4HZXA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: write tcp 5.161.89.252:8238->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980403213 (263h58m22.711670863s) [noise=true , address=77.90.31.19:4841 , piece_id=MY2CTXR6M4Z7JDC5BE3HENPMUOXXSVX22PNG236QOH5F3552PFCQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980547475 (263h58m22.711815124s) [noise=true , address=45.91.116.6:4973 , piece_id=MP533ZUEZDRXYTQPWKM5URFX6BB3WPGAPFHFMUJQVNBZFPTJHICA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 5.161.89.252:29404->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980632083 (263h58m22.711899743s) [noise=true , address=77.90.31.12:4592 , piece_id=LYEZMNLTZVT4UQHZL53CGXZXPHLPJPO34EMLPPRARNVULCOQPXKQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980736129 (263h58m22.712003779s) [seg_pos={0 0}, upload=274520]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:41.997898729 (263h58m22.729166379s) [upload=274520]: Done uploading segments... Aug 19 17:24:41.997919457 (263h58m22.729187108s) [upload=274520]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 539.730296ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....XX..............................................X...X..] duration: 419.387409ms FAIL (2025-08-19 17:26:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 419.424979ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X....................................X.X.X..] duration: 498.274114ms FAIL (2025-08-19 17:25:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 19 17:24:12.613365846 (263h57m53.344633476s) [upload=274521]: Starting upload Aug 19 17:24:12.613406413 (263h57m53.344674043s) [upload=274521]: Done starting upload Aug 19 17:24:12.613493606 (263h57m53.344761226s) [upload=274521]: Uploading segments... Aug 19 17:24:12.613734899 (263h57m53.345002529s) [upload=274521]: Got next segment. Inline: false Aug 19 17:24:12.613764896 (263h57m53.345032526s) [upload=274521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:12.613769284 (263h57m53.345036914s) [upload=274521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:12.651951109 (263h57m53.383218739s) [seg_pos={0 0}, upload=274521]: Begin upload segment... Aug 19 17:24:12.651990573 (263h57m53.383258193s) [seg_pos={0 0}, upload=274521]: Done begin upload segment. Aug 19 17:24:12.652011803 (263h57m53.383279433s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652035357 (263h57m53.383302987s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652327476 (263h57m53.383595106s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652383030 (263h57m53.38365066s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652565463 (263h57m53.383833093s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652676883 (263h57m53.383944513s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652812508 (263h57m53.384080138s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.652949715 (263h57m53.384217345s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653091241 (263h57m53.384358871s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653251271 (263h57m53.384518892s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653305264 (263h57m53.384572884s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653509577 (263h57m53.384777207s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653518354 (263h57m53.384785984s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653743006 (263h57m53.385010626s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653788721 (263h57m53.385056351s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.653985512 (263h57m53.385253132s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654086841 (263h57m53.385354471s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.654373080 (263h57m53.3856407s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Uploading piece... Aug 19 17:24:12.735474331 (263h57m53.466741961s) [upload=274521]: Next returned nil segment Aug 19 17:24:12.735486304 (263h57m53.466753934s) [upload=274521]: Waiting for error group managing segments... Aug 19 17:24:12.756883830 (263h57m53.48815145s) [noise=true , address=104.36.82.18:4381 , piece_id=RKK7ZUWNENGDCVUT5WG7BDDWV27PI3HXXKLAIFBTBCURFEMAYWPA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.766603935 (263h57m53.497871585s) [noise=true , address=104.36.82.45:4911 , piece_id=ZXUDCVWZU52QB5GD5CRK3KECCMLBNDSJSAE3ODVHR7CYRDI4TDGA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.809581250 (263h57m53.540859941s) [noise=true , address=104.36.82.37:4654 , piece_id=YHZ2ZZLFZIL6WU6SQFABTJYWLO7Q3IKQ6YRHV5O66T2WDULFX4LQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:12.908061427 (263h57m53.639329057s) [noise=true , address=79.127.201.215:4354 , piece_id=ILHYGB4D4OC5ZLMIQMOAP2VBUTPPZP73GSAWTWBVQCYD7UFCJ5ZA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.147463951 (263h57m53.878731601s) [noise=true , address=79.127.205.246:4258 , piece_id=62LLK7AA6RSVT6DGM6OBIMQVSTSVL7CKVOLSA4SA4UXBHRWINPOQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:13.364439955 (263h57m54.095707575s) [noise=true , address=64.49.13.26:4950 , piece_id=DF2W5SXMRPYVOO3VYRBOUICKMLALFEUO44RIRXHPXBINUUQ2KOHQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:51.706547376 (263h58m32.437815006s) [noise=true , address=77.90.31.12:4610 , piece_id=KGPE5FMXNSZBCXFT2RICGZBMN6NBHCDHNL5DAMJ4ERCKJTNHJYRA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:52.370283683 (263h58m33.101551303s) [noise=true , address=77.90.31.14:4672 , piece_id=S7LZJKYJYODOSPDQTY3IGUZDAUEV3FP2NOJ4YDJYPLP5OLGXIYXQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.816229069 (263h58m36.547496699s) [noise=true , address=77.90.31.18:4837 , piece_id=N7L4YWBPVUTRQGYK7DP6YRFXUTV4PPCK6ZP3PU3WTVYJOVKEEFQQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:24:55.944464809 (263h58m36.675732439s) [noise=true , address=77.90.31.16:4747 , piece_id=O7DENBKJSUM475ACFXOZ76ZNHPTKPKNVWRI5EODS4NSV7QFK5JCA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.083334942 (263h58m48.814602572s) [noise=true , address=45.91.116.10:4089 , piece_id=3COZWVBKUODK4BY2ZRZHWDF5Z2LQRJMT7WWDEEK5U24R7GHMEMFA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:08.405488480 (263h58m49.136756101s) [noise=true , address=45.91.116.11:4130 , piece_id=I3YU33P2NBUHDO25RRJAZA7Q4PBEKHGS6L2LS6QU4PZ54N5FYRWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.160489383 (263h58m52.891757013s) [noise=true , address=45.91.116.8:4046 , piece_id=H5JXXHFSR3A6FCZHHGZMQNYNSHME32X2SJ4B2CFRPCJFFYH4XVGQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: <nil> Aug 19 17:25:12.595037822 (263h58m53.326305452s) [noise=true , address=45.91.116.14:4213 , piece_id=M44PJHBENOPUNW7Y36ALXKWWSYSNKIVCWMMRPUSTQARFIUBHSSFQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595089570 (263h58m53.326357189s) [noise=true , address=45.91.116.12:4250 , piece_id=T3LKOF56LCVHNCFEFEPJWOW5SV6CYPASBLIMOEEMG7SCDITQ6DLQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 5.161.89.252:56044->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595102303 (263h58m53.326369933s) [noise=true , address=45.91.116.4:4950 , piece_id=AHBPJ5MQGTIZATXP6WMQK7BB3FDWKDWNONTBVICHCBJLIK6VIDTA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: rpc: dial tcp 45.91.116.4:4950: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595112262 (263h58m53.326379892s) [noise=true , address=77.90.31.11:4574 , piece_id=TXNGYH7XVLWTNFITEVSI4LY23T6WBQT4OLFGGXWPLPY57KW4KZWA, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 5.161.89.252:9456->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595145234 (263h58m53.326412864s) [noise=true , address=45.91.116.13:4182 , piece_id=GHBD43AKTIKHGLNGIIIBYDIT3RLEWI226TWIWGH5TVSOQHLQ4QIQ, seg_pos={0 0}, upload=274521]: Done uploading piece. err: ecclient: upload failed (node:1gnJixpV2uZqEMTxi22y1EAzKCHXaQeBWakhT6AcEkrENWT8CT, address:45.91.116.13:4182): protocol: write tcp 5.161.89.252:9502->45.91.116.13:4182: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:25:12.595163289 (263h58m53.326430909s) [seg_pos={0 0}, upload=274521]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 19 17:25:12.636371896 (263h58m53.367639527s) [upload=274521]: Done uploading segments... Aug 19 17:25:12.636391424 (263h58m53.367659054s) [upload=274521]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 498.292169ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X....X..............................................X....X..] duration: 486.60269ms FAIL (2025-08-19 17:25:45 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== [seg_pos={0 0}, upload=274520]: Done begin upload segment. Aug 19 17:24:12.024531859 (263h57m52.755799509s) [noise=true , address=104.36.82.9:4007 , piece_id=YEAQAGKXBODCVNIAG2S7TT5MP3TKTC5PAWGA7CIIBEW5DYS7UQCQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024547178 (263h57m52.755814828s) [noise=true , address=45.91.116.8:4045 , piece_id=X2HZK7QLI7RRZMF6WYSHK4TYUCPWMF7XRY6KVWKG7SOMGZK4HZXA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024953521 (263h57m52.756221171s) [noise=true , address=77.90.31.11:4554 , piece_id=OQYJJPQRLA64VFXA5VBWDKJ73Q2YMLTRW3UXJVNBQ3BLCZ55TZ5Q, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.024997523 (263h57m52.756265174s) [noise=true , address=45.91.116.6:4973 , piece_id=MP533ZUEZDRXYTQPWKM5URFX6BB3WPGAPFHFMUJQVNBZFPTJHICA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025367439 (263h57m52.756635089s) [noise=true , address=77.90.31.19:4841 , piece_id=MY2CTXR6M4Z7JDC5BE3HENPMUOXXSVX22PNG236QOH5F3552PFCQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025414347 (263h57m52.756681997s) [noise=true , address=104.36.82.2:4729 , piece_id=LY2SWRDWXQHES5FTAF5CP6KCNVCY6FHIBKJNLOCEFYSBNBPIPNIA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025761920 (263h57m52.75702958s) [noise=true , address=104.36.82.7:4909 , piece_id=QB6Z7VE6VFC5L3BNINLJTO53YACDUGNBL6U7WUHKSG5HAFVSXAJQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.025888298 (263h57m52.757155948s) [noise=true , address=77.90.31.14:4661 , piece_id=WDR237S6GBHWJGAO2L7M5YASTXX6QOLMIMDZM2D57PQHZA7BYB2Q, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026138559 (263h57m52.757406208s) [noise=true , address=77.90.31.12:4592 , piece_id=LYEZMNLTZVT4UQHZL53CGXZXPHLPJPO34EMLPPRARNVULCOQPXKQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026283681 (263h57m52.75755133s) [noise=true , address=104.36.82.5:4837 , piece_id=AM2AXREOT4ZII77PAWUE76BM6AR4YGI3XF5CGUWRKEE5R5FTDIFA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026491962 (263h57m52.757759613s) [noise=true , address=45.91.116.14:4189 , piece_id=QVER7IKF6T66GHOKDYVUVAYBF4OWORQV3IPKXKTREBYXOLKUI6HA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026640120 (263h57m52.757907771s) [noise=true , address=45.91.116.14:4190 , piece_id=3AZCPPFTZFZ7ANINXPAIR433CKJUXEVDB7FYJIQDLHMYIMRFTGKQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026842701 (263h57m52.758110352s) [noise=true , address=77.90.31.13:4628 , piece_id=B2BESUZXB37NOFSOEUPCUXX3AMEZ44S62HIKN7K4P2FD65CPEDPQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.026995178 (263h57m52.758262828s) [noise=true , address=104.36.82.7:4912 , piece_id=ADZS3KGOOF4HPEYRXYTTS7WHDRKRBGRIKUKCORRSAENYPHXMEGHA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.027210582 (263h57m52.758478232s) [noise=true , address=104.36.82.13:4185 , piece_id=G6V7LQWJ3UC4PMMPBZWE67KWQXQENEEO4O4763VEGVGEVIDJ5FEA, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.027858310 (263h57m52.75912596s) [noise=true , address=45.91.116.4:4939 , piece_id=BTZOJWID4G2ADAZKPQ6PW7SMV42R2FM756KFMCBSBAMFMJDFULWQ, seg_pos={0 0}, upload=274520]: Uploading piece... Aug 19 17:24:12.143746757 (263h57m52.875014407s) [upload=274520]: Next returned nil segment Aug 19 17:24:12.143775502 (263h57m52.875043152s) [upload=274520]: Waiting for error group managing segments... Aug 19 17:24:12.157847553 (263h57m52.889115213s) [noise=true , address=104.36.82.13:4185 , piece_id=G6V7LQWJ3UC4PMMPBZWE67KWQXQENEEO4O4763VEGVGEVIDJ5FEA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.158547338 (263h57m52.889814989s) [noise=true , address=104.36.82.5:4837 , piece_id=AM2AXREOT4ZII77PAWUE76BM6AR4YGI3XF5CGUWRKEE5R5FTDIFA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.158808157 (263h57m52.890075818s) [noise=true , address=104.36.82.2:4729 , piece_id=LY2SWRDWXQHES5FTAF5CP6KCNVCY6FHIBKJNLOCEFYSBNBPIPNIA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.159159038 (263h57m52.890426688s) [noise=true , address=104.36.82.7:4912 , piece_id=ADZS3KGOOF4HPEYRXYTTS7WHDRKRBGRIKUKCORRSAENYPHXMEGHA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.160381044 (263h57m52.891648694s) [noise=true , address=104.36.82.7:4909 , piece_id=QB6Z7VE6VFC5L3BNINLJTO53YACDUGNBL6U7WUHKSG5HAFVSXAJQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:12.163470309 (263h57m52.894737959s) [noise=true , address=104.36.82.9:4007 , piece_id=YEAQAGKXBODCVNIAG2S7TT5MP3TKTC5PAWGA7CIIBEW5DYS7UQCQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: <nil> Aug 19 17:24:41.979520414 (263h58m22.710788064s) [noise=true , address=77.90.31.13:4628 , piece_id=B2BESUZXB37NOFSOEUPCUXX3AMEZ44S62HIKN7K4P2FD65CPEDPQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 5.161.89.252:36766->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980102297 (263h58m22.711369958s) [noise=true , address=45.91.116.14:4190 , piece_id=3AZCPPFTZFZ7ANINXPAIR433CKJUXEVDB7FYJIQDLHMYIMRFTGKQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:21646->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980158464 (263h58m22.711426114s) [noise=true , address=45.91.116.4:4939 , piece_id=BTZOJWID4G2ADAZKPQ6PW7SMV42R2FM756KFMCBSBAMFMJDFULWQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 5.161.89.252:51578->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980204240 (263h58m22.71147189s) [noise=true , address=77.90.31.11:4554 , piece_id=OQYJJPQRLA64VFXA5VBWDKJ73Q2YMLTRW3UXJVNBQ3BLCZ55TZ5Q, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 5.161.89.252:9778->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980249345 (263h58m22.711517004s) [noise=true , address=77.90.31.14:4661 , piece_id=WDR237S6GBHWJGAO2L7M5YASTXX6QOLMIMDZM2D57PQHZA7BYB2Q, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:22472->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980292254 (263h58m22.711559905s) [noise=true , address=45.91.116.14:4189 , piece_id=QVER7IKF6T66GHOKDYVUVAYBF4OWORQV3IPKXKTREBYXOLKUI6HA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: read tcp 5.161.89.252:34124->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980365091 (263h58m22.711632752s) [noise=true , address=45.91.116.8:4045 , piece_id=X2HZK7QLI7RRZMF6WYSHK4TYUCPWMF7XRY6KVWKG7SOMGZK4HZXA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: write tcp 5.161.89.252:8238->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980403213 (263h58m22.711670863s) [noise=true , address=77.90.31.19:4841 , piece_id=MY2CTXR6M4Z7JDC5BE3HENPMUOXXSVX22PNG236QOH5F3552PFCQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980547475 (263h58m22.711815124s) [noise=true , address=45.91.116.6:4973 , piece_id=MP533ZUEZDRXYTQPWKM5URFX6BB3WPGAPFHFMUJQVNBZFPTJHICA, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 5.161.89.252:29404->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980632083 (263h58m22.711899743s) [noise=true , address=77.90.31.12:4592 , piece_id=LYEZMNLTZVT4UQHZL53CGXZXPHLPJPO34EMLPPRARNVULCOQPXKQ, seg_pos={0 0}, upload=274520]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:41.980736129 (263h58m22.712003779s) [seg_pos={0 0}, upload=274520]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:41.997898729 (263h58m22.729166379s) [upload=274520]: Done uploading segments... Aug 19 17:24:41.997919457 (263h58m22.729187108s) [upload=274520]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 486.626735ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)