Orbiter History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.243324142s FAIL (2025-09-22 05:54:45 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.243349521s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..................X........................................] duration: 337.302713ms FAIL (2025-09-22 05:54:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment... Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment. Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments... Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments... Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 337.319194ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 4.910811743s FAIL (2025-09-22 05:54:38 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.910840376s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.X.....X.......X.X...X..XX.................................] duration: 413.159745ms FAIL (2025-09-22 05:54:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:31:26.835749533 (567h23m2.188534662s) [upload=499337]: Starting upload Sep 22 05:31:26.835842508 (567h23m2.188627636s) [upload=499337]: Done starting upload Sep 22 05:31:26.836008360 (567h23m2.188793499s) [upload=499337]: Uploading segments... Sep 22 05:31:26.836536982 (567h23m2.18932211s) [upload=499337]: Got next segment. Inline: false Sep 22 05:31:26.836604177 (567h23m2.189389316s) [upload=499337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:31:26.836610950 (567h23m2.189396078s) [upload=499337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:31:26.856276449 (567h23m2.209061577s) [seg_pos={0 0}, upload=499337]: Begin upload segment... Sep 22 05:31:26.856314771 (567h23m2.209099899s) [seg_pos={0 0}, upload=499337]: Done begin upload segment. Sep 22 05:31:26.856372709 (567h23m2.209157838s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856423275 (567h23m2.209208413s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856888909 (567h23m2.209674037s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856930226 (567h23m2.209715354s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857336809 (567h23m2.210121948s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857426228 (567h23m2.210211355s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857842308 (567h23m2.210627436s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857850002 (567h23m2.210635131s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858275311 (567h23m2.211060449s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858418950 (567h23m2.211204089s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858708293 (567h23m2.211493421s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858912537 (567h23m2.211697665s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859330581 (567h23m2.212115709s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859801715 (567h23m2.212586844s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.860342481 (567h23m2.213127609s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.861314445 (567h23m2.214099573s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:27.009344801 (567h23m2.362129929s) [upload=499337]: Next returned nil segment Sep 22 05:31:27.009357354 (567h23m2.362142472s) [upload=499337]: Waiting for error group managing segments... Sep 22 05:31:27.018549517 (567h23m2.371334634s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.019540115 (567h23m2.372325234s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020276638 (567h23m2.373061756s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020786695 (567h23m2.373571803s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021133024 (567h23m2.373918133s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021696293 (567h23m2.37448141s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021726819 (567h23m2.374511927s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022237469 (567h23m2.375022586s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022647758 (567h23m2.375432876s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022916483 (567h23m2.3757016s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023451627 (567h23m2.376236745s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023639529 (567h23m2.376424648s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:30.479420128 (567h23m5.832205236s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; manager closed: conn closed: op Sep 22 05:31:56.821587941 (567h23m32.174373059s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821645730 (567h23m32.174430847s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821723946 (567h23m32.174509064s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; context deadline exceeded Sep 22 05:31:56.821747761 (567h23m32.174532879s) [seg_pos={0 0}, upload=499337]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:31:56.838365960 (567h23m32.191151059s) [upload=499337]: Done uploading segments... Sep 22 05:31:56.838377933 (567h23m32.191163031s) [upload=499337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 413.185332ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 4.995821534s FAIL (2025-09-22 05:54:04 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.995842443s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.....X....XXX..........X.X.......XXX....X................] duration: 421.878327ms FAIL (2025-09-22 05:54:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 421.895369ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.180273749s FAIL (2025-09-22 05:53:51 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.18029535s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..................X.......................................] duration: 405.418337ms FAIL (2025-09-22 05:53:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment... Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment. Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments... Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments... Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 405.440939ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX.X.....X.......X.X...X..XX................................] duration: 638.442412ms FAIL (2025-09-22 05:53:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:31:26.835749533 (567h23m2.188534662s) [upload=499337]: Starting upload Sep 22 05:31:26.835842508 (567h23m2.188627636s) [upload=499337]: Done starting upload Sep 22 05:31:26.836008360 (567h23m2.188793499s) [upload=499337]: Uploading segments... Sep 22 05:31:26.836536982 (567h23m2.18932211s) [upload=499337]: Got next segment. Inline: false Sep 22 05:31:26.836604177 (567h23m2.189389316s) [upload=499337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:31:26.836610950 (567h23m2.189396078s) [upload=499337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:31:26.856276449 (567h23m2.209061577s) [seg_pos={0 0}, upload=499337]: Begin upload segment... Sep 22 05:31:26.856314771 (567h23m2.209099899s) [seg_pos={0 0}, upload=499337]: Done begin upload segment. Sep 22 05:31:26.856372709 (567h23m2.209157838s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856423275 (567h23m2.209208413s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856888909 (567h23m2.209674037s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856930226 (567h23m2.209715354s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857336809 (567h23m2.210121948s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857426228 (567h23m2.210211355s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857842308 (567h23m2.210627436s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857850002 (567h23m2.210635131s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858275311 (567h23m2.211060449s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858418950 (567h23m2.211204089s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858708293 (567h23m2.211493421s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858912537 (567h23m2.211697665s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859330581 (567h23m2.212115709s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859801715 (567h23m2.212586844s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.860342481 (567h23m2.213127609s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.861314445 (567h23m2.214099573s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:27.009344801 (567h23m2.362129929s) [upload=499337]: Next returned nil segment Sep 22 05:31:27.009357354 (567h23m2.362142472s) [upload=499337]: Waiting for error group managing segments... Sep 22 05:31:27.018549517 (567h23m2.371334634s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.019540115 (567h23m2.372325234s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020276638 (567h23m2.373061756s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020786695 (567h23m2.373571803s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021133024 (567h23m2.373918133s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021696293 (567h23m2.37448141s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021726819 (567h23m2.374511927s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022237469 (567h23m2.375022586s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022647758 (567h23m2.375432876s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022916483 (567h23m2.3757016s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023451627 (567h23m2.376236745s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023639529 (567h23m2.376424648s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:30.479420128 (567h23m5.832205236s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; manager closed: conn closed: op Sep 22 05:31:56.821587941 (567h23m32.174373059s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821645730 (567h23m32.174430847s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821723946 (567h23m32.174509064s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; context deadline exceeded Sep 22 05:31:56.821747761 (567h23m32.174532879s) [seg_pos={0 0}, upload=499337]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:31:56.838365960 (567h23m32.191151059s) [upload=499337]: Done uploading segments... Sep 22 05:31:56.838377933 (567h23m32.191163031s) [upload=499337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 638.464494ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..................X......................................] duration: 638.613123ms FAIL (2025-09-22 05:53:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment... Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment. Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments... Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments... Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 638.632949ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.014804964s FAIL (2025-09-22 05:53:19 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.014821796s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.165416631s FAIL (2025-09-22 05:53:14 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.165436018s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X.....X....XXX..........X.X.......XXX....X...............] duration: 458.918148ms FAIL (2025-09-22 05:53:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 459.032603ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXX.X.....X.......X.X...X..XX...............................] duration: 792.256249ms FAIL (2025-09-22 05:52:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:31:26.835749533 (567h23m2.188534662s) [upload=499337]: Starting upload Sep 22 05:31:26.835842508 (567h23m2.188627636s) [upload=499337]: Done starting upload Sep 22 05:31:26.836008360 (567h23m2.188793499s) [upload=499337]: Uploading segments... Sep 22 05:31:26.836536982 (567h23m2.18932211s) [upload=499337]: Got next segment. Inline: false Sep 22 05:31:26.836604177 (567h23m2.189389316s) [upload=499337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:31:26.836610950 (567h23m2.189396078s) [upload=499337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:31:26.856276449 (567h23m2.209061577s) [seg_pos={0 0}, upload=499337]: Begin upload segment... Sep 22 05:31:26.856314771 (567h23m2.209099899s) [seg_pos={0 0}, upload=499337]: Done begin upload segment. Sep 22 05:31:26.856372709 (567h23m2.209157838s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856423275 (567h23m2.209208413s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856888909 (567h23m2.209674037s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856930226 (567h23m2.209715354s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857336809 (567h23m2.210121948s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857426228 (567h23m2.210211355s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857842308 (567h23m2.210627436s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857850002 (567h23m2.210635131s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858275311 (567h23m2.211060449s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858418950 (567h23m2.211204089s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858708293 (567h23m2.211493421s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858912537 (567h23m2.211697665s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859330581 (567h23m2.212115709s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859801715 (567h23m2.212586844s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.860342481 (567h23m2.213127609s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.861314445 (567h23m2.214099573s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:27.009344801 (567h23m2.362129929s) [upload=499337]: Next returned nil segment Sep 22 05:31:27.009357354 (567h23m2.362142472s) [upload=499337]: Waiting for error group managing segments... Sep 22 05:31:27.018549517 (567h23m2.371334634s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.019540115 (567h23m2.372325234s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020276638 (567h23m2.373061756s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020786695 (567h23m2.373571803s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021133024 (567h23m2.373918133s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021696293 (567h23m2.37448141s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021726819 (567h23m2.374511927s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022237469 (567h23m2.375022586s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022647758 (567h23m2.375432876s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022916483 (567h23m2.3757016s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023451627 (567h23m2.376236745s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023639529 (567h23m2.376424648s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:30.479420128 (567h23m5.832205236s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; manager closed: conn closed: op Sep 22 05:31:56.821587941 (567h23m32.174373059s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821645730 (567h23m32.174430847s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821723946 (567h23m32.174509064s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; context deadline exceeded Sep 22 05:31:56.821747761 (567h23m32.174532879s) [seg_pos={0 0}, upload=499337]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:31:56.838365960 (567h23m32.191151059s) [upload=499337]: Done uploading segments... Sep 22 05:31:56.838377933 (567h23m32.191163031s) [upload=499337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 792.272639ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..X..................X.....................................] duration: 538.61797ms FAIL (2025-09-22 05:52:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment... Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment. Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments... Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments... Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 538.63932ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.265741461s FAIL (2025-09-22 05:52:53 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.265768101s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000904792s FAIL (2025-09-22 05:52:40 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 30.000921323s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..X..................X....................................] duration: 408.410023ms FAIL (2025-09-22 05:52:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment... Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment. Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments... Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments... Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 408.441653ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001134916s FAIL (2025-09-22 05:52:19 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 30.001157088s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.XXX.X.....X.......X.X...X..XX..............................] duration: 718.767719ms FAIL (2025-09-22 05:52:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:31:26.835749533 (567h23m2.188534662s) [upload=499337]: Starting upload Sep 22 05:31:26.835842508 (567h23m2.188627636s) [upload=499337]: Done starting upload Sep 22 05:31:26.836008360 (567h23m2.188793499s) [upload=499337]: Uploading segments... Sep 22 05:31:26.836536982 (567h23m2.18932211s) [upload=499337]: Got next segment. Inline: false Sep 22 05:31:26.836604177 (567h23m2.189389316s) [upload=499337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:31:26.836610950 (567h23m2.189396078s) [upload=499337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:31:26.856276449 (567h23m2.209061577s) [seg_pos={0 0}, upload=499337]: Begin upload segment... Sep 22 05:31:26.856314771 (567h23m2.209099899s) [seg_pos={0 0}, upload=499337]: Done begin upload segment. Sep 22 05:31:26.856372709 (567h23m2.209157838s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856423275 (567h23m2.209208413s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856888909 (567h23m2.209674037s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856930226 (567h23m2.209715354s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857336809 (567h23m2.210121948s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857426228 (567h23m2.210211355s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857842308 (567h23m2.210627436s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857850002 (567h23m2.210635131s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858275311 (567h23m2.211060449s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858418950 (567h23m2.211204089s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858708293 (567h23m2.211493421s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858912537 (567h23m2.211697665s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859330581 (567h23m2.212115709s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859801715 (567h23m2.212586844s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.860342481 (567h23m2.213127609s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.861314445 (567h23m2.214099573s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:27.009344801 (567h23m2.362129929s) [upload=499337]: Next returned nil segment Sep 22 05:31:27.009357354 (567h23m2.362142472s) [upload=499337]: Waiting for error group managing segments... Sep 22 05:31:27.018549517 (567h23m2.371334634s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.019540115 (567h23m2.372325234s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020276638 (567h23m2.373061756s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020786695 (567h23m2.373571803s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021133024 (567h23m2.373918133s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021696293 (567h23m2.37448141s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021726819 (567h23m2.374511927s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022237469 (567h23m2.375022586s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022647758 (567h23m2.375432876s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022916483 (567h23m2.3757016s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023451627 (567h23m2.376236745s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023639529 (567h23m2.376424648s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:30.479420128 (567h23m5.832205236s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; manager closed: conn closed: op Sep 22 05:31:56.821587941 (567h23m32.174373059s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821645730 (567h23m32.174430847s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821723946 (567h23m32.174509064s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; context deadline exceeded Sep 22 05:31:56.821747761 (567h23m32.174532879s) [seg_pos={0 0}, upload=499337]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:31:56.838365960 (567h23m32.191151059s) [upload=499337]: Done uploading segments... Sep 22 05:31:56.838377933 (567h23m32.191163031s) [upload=499337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 718.785743ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X...X.....X....XXX..........X.X.......XXX....X..............] duration: 389.008623ms FAIL (2025-09-22 05:51:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 389.075838ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..XXX.X.....X.......X.X...X..XX.............................] duration: 768.790974ms FAIL (2025-09-22 05:51:54 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:31:26.835749533 (567h23m2.188534662s) [upload=499337]: Starting upload Sep 22 05:31:26.835842508 (567h23m2.188627636s) [upload=499337]: Done starting upload Sep 22 05:31:26.836008360 (567h23m2.188793499s) [upload=499337]: Uploading segments... Sep 22 05:31:26.836536982 (567h23m2.18932211s) [upload=499337]: Got next segment. Inline: false Sep 22 05:31:26.836604177 (567h23m2.189389316s) [upload=499337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:31:26.836610950 (567h23m2.189396078s) [upload=499337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:31:26.856276449 (567h23m2.209061577s) [seg_pos={0 0}, upload=499337]: Begin upload segment... Sep 22 05:31:26.856314771 (567h23m2.209099899s) [seg_pos={0 0}, upload=499337]: Done begin upload segment. Sep 22 05:31:26.856372709 (567h23m2.209157838s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856423275 (567h23m2.209208413s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856888909 (567h23m2.209674037s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856930226 (567h23m2.209715354s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857336809 (567h23m2.210121948s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857426228 (567h23m2.210211355s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857842308 (567h23m2.210627436s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857850002 (567h23m2.210635131s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858275311 (567h23m2.211060449s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858418950 (567h23m2.211204089s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858708293 (567h23m2.211493421s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858912537 (567h23m2.211697665s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859330581 (567h23m2.212115709s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859801715 (567h23m2.212586844s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.860342481 (567h23m2.213127609s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.861314445 (567h23m2.214099573s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:27.009344801 (567h23m2.362129929s) [upload=499337]: Next returned nil segment Sep 22 05:31:27.009357354 (567h23m2.362142472s) [upload=499337]: Waiting for error group managing segments... Sep 22 05:31:27.018549517 (567h23m2.371334634s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.019540115 (567h23m2.372325234s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020276638 (567h23m2.373061756s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020786695 (567h23m2.373571803s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021133024 (567h23m2.373918133s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021696293 (567h23m2.37448141s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021726819 (567h23m2.374511927s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022237469 (567h23m2.375022586s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022647758 (567h23m2.375432876s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022916483 (567h23m2.3757016s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023451627 (567h23m2.376236745s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023639529 (567h23m2.376424648s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:30.479420128 (567h23m5.832205236s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; manager closed: conn closed: op Sep 22 05:31:56.821587941 (567h23m32.174373059s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821645730 (567h23m32.174430847s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821723946 (567h23m32.174509064s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; context deadline exceeded Sep 22 05:31:56.821747761 (567h23m32.174532879s) [seg_pos={0 0}, upload=499337]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:31:56.838365960 (567h23m32.191151059s) [upload=499337]: Done uploading segments... Sep 22 05:31:56.838377933 (567h23m32.191163031s) [upload=499337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 768.812004ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.X..X..................X...................................] duration: 626.720032ms FAIL (2025-09-22 05:51:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment... Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment. Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments... Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments... Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 626.73983ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...XXX.X.....X.......X.X...X..XX............................] duration: 541.471995ms FAIL (2025-09-22 05:50:50 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:31:26.835749533 (567h23m2.188534662s) [upload=499337]: Starting upload Sep 22 05:31:26.835842508 (567h23m2.188627636s) [upload=499337]: Done starting upload Sep 22 05:31:26.836008360 (567h23m2.188793499s) [upload=499337]: Uploading segments... Sep 22 05:31:26.836536982 (567h23m2.18932211s) [upload=499337]: Got next segment. Inline: false Sep 22 05:31:26.836604177 (567h23m2.189389316s) [upload=499337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:31:26.836610950 (567h23m2.189396078s) [upload=499337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:31:26.856276449 (567h23m2.209061577s) [seg_pos={0 0}, upload=499337]: Begin upload segment... Sep 22 05:31:26.856314771 (567h23m2.209099899s) [seg_pos={0 0}, upload=499337]: Done begin upload segment. Sep 22 05:31:26.856372709 (567h23m2.209157838s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856423275 (567h23m2.209208413s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856888909 (567h23m2.209674037s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856930226 (567h23m2.209715354s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857336809 (567h23m2.210121948s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857426228 (567h23m2.210211355s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857842308 (567h23m2.210627436s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857850002 (567h23m2.210635131s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858275311 (567h23m2.211060449s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858418950 (567h23m2.211204089s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858708293 (567h23m2.211493421s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858912537 (567h23m2.211697665s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859330581 (567h23m2.212115709s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859801715 (567h23m2.212586844s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.860342481 (567h23m2.213127609s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.861314445 (567h23m2.214099573s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:27.009344801 (567h23m2.362129929s) [upload=499337]: Next returned nil segment Sep 22 05:31:27.009357354 (567h23m2.362142472s) [upload=499337]: Waiting for error group managing segments... Sep 22 05:31:27.018549517 (567h23m2.371334634s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.019540115 (567h23m2.372325234s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020276638 (567h23m2.373061756s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020786695 (567h23m2.373571803s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021133024 (567h23m2.373918133s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021696293 (567h23m2.37448141s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021726819 (567h23m2.374511927s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022237469 (567h23m2.375022586s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022647758 (567h23m2.375432876s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022916483 (567h23m2.3757016s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023451627 (567h23m2.376236745s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023639529 (567h23m2.376424648s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:30.479420128 (567h23m5.832205236s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; manager closed: conn closed: op Sep 22 05:31:56.821587941 (567h23m32.174373059s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821645730 (567h23m32.174430847s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821723946 (567h23m32.174509064s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; context deadline exceeded Sep 22 05:31:56.821747761 (567h23m32.174532879s) [seg_pos={0 0}, upload=499337]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:31:56.838365960 (567h23m32.191151059s) [upload=499337]: Done uploading segments... Sep 22 05:31:56.838377933 (567h23m32.191163031s) [upload=499337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 541.504748ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.X..X..................X..................................] duration: 373.216669ms FAIL (2025-09-22 05:50:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment... Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment. Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments... Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments... Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 373.237526ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X...X.....X....XXX..........X.X.......XXX....X.............] duration: 395.023912ms FAIL (2025-09-22 05:50:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 395.037356ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.X..X..................X.................................] duration: 363.867844ms FAIL (2025-09-22 05:50:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment... Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment. Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments... Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments... Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 363.88653ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X...X.....X....XXX..........X.X.......XXX....X............] duration: 609.567375ms FAIL (2025-09-22 05:49:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 609.59637ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....XXX.X.....X.......X.X...X..XX...........................] duration: 772.502269ms FAIL (2025-09-22 05:49:49 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:31:26.835749533 (567h23m2.188534662s) [upload=499337]: Starting upload Sep 22 05:31:26.835842508 (567h23m2.188627636s) [upload=499337]: Done starting upload Sep 22 05:31:26.836008360 (567h23m2.188793499s) [upload=499337]: Uploading segments... Sep 22 05:31:26.836536982 (567h23m2.18932211s) [upload=499337]: Got next segment. Inline: false Sep 22 05:31:26.836604177 (567h23m2.189389316s) [upload=499337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:31:26.836610950 (567h23m2.189396078s) [upload=499337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:31:26.856276449 (567h23m2.209061577s) [seg_pos={0 0}, upload=499337]: Begin upload segment... Sep 22 05:31:26.856314771 (567h23m2.209099899s) [seg_pos={0 0}, upload=499337]: Done begin upload segment. Sep 22 05:31:26.856372709 (567h23m2.209157838s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856423275 (567h23m2.209208413s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856888909 (567h23m2.209674037s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856930226 (567h23m2.209715354s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857336809 (567h23m2.210121948s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857426228 (567h23m2.210211355s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857842308 (567h23m2.210627436s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857850002 (567h23m2.210635131s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858275311 (567h23m2.211060449s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858418950 (567h23m2.211204089s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858708293 (567h23m2.211493421s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858912537 (567h23m2.211697665s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859330581 (567h23m2.212115709s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859801715 (567h23m2.212586844s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.860342481 (567h23m2.213127609s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.861314445 (567h23m2.214099573s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:27.009344801 (567h23m2.362129929s) [upload=499337]: Next returned nil segment Sep 22 05:31:27.009357354 (567h23m2.362142472s) [upload=499337]: Waiting for error group managing segments... Sep 22 05:31:27.018549517 (567h23m2.371334634s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.019540115 (567h23m2.372325234s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020276638 (567h23m2.373061756s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020786695 (567h23m2.373571803s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021133024 (567h23m2.373918133s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021696293 (567h23m2.37448141s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021726819 (567h23m2.374511927s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022237469 (567h23m2.375022586s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022647758 (567h23m2.375432876s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022916483 (567h23m2.3757016s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023451627 (567h23m2.376236745s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023639529 (567h23m2.376424648s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:30.479420128 (567h23m5.832205236s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; manager closed: conn closed: op Sep 22 05:31:56.821587941 (567h23m32.174373059s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821645730 (567h23m32.174430847s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821723946 (567h23m32.174509064s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; context deadline exceeded Sep 22 05:31:56.821747761 (567h23m32.174532879s) [seg_pos={0 0}, upload=499337]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:31:56.838365960 (567h23m32.191151059s) [upload=499337]: Done uploading segments... Sep 22 05:31:56.838377933 (567h23m32.191163031s) [upload=499337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 772.523249ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.X..X..................X................................] duration: 343.728722ms FAIL (2025-09-22 05:49:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment... Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment. Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments... Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments... Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 343.747277ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....XXX.X.....X.......X.X...X..XX..........................] duration: 437.591194ms FAIL (2025-09-22 05:49:11 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:31:26.835749533 (567h23m2.188534662s) [upload=499337]: Starting upload Sep 22 05:31:26.835842508 (567h23m2.188627636s) [upload=499337]: Done starting upload Sep 22 05:31:26.836008360 (567h23m2.188793499s) [upload=499337]: Uploading segments... Sep 22 05:31:26.836536982 (567h23m2.18932211s) [upload=499337]: Got next segment. Inline: false Sep 22 05:31:26.836604177 (567h23m2.189389316s) [upload=499337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:31:26.836610950 (567h23m2.189396078s) [upload=499337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:31:26.856276449 (567h23m2.209061577s) [seg_pos={0 0}, upload=499337]: Begin upload segment... Sep 22 05:31:26.856314771 (567h23m2.209099899s) [seg_pos={0 0}, upload=499337]: Done begin upload segment. Sep 22 05:31:26.856372709 (567h23m2.209157838s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856423275 (567h23m2.209208413s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856888909 (567h23m2.209674037s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856930226 (567h23m2.209715354s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857336809 (567h23m2.210121948s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857426228 (567h23m2.210211355s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857842308 (567h23m2.210627436s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857850002 (567h23m2.210635131s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858275311 (567h23m2.211060449s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858418950 (567h23m2.211204089s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858708293 (567h23m2.211493421s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858912537 (567h23m2.211697665s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859330581 (567h23m2.212115709s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859801715 (567h23m2.212586844s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.860342481 (567h23m2.213127609s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.861314445 (567h23m2.214099573s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:27.009344801 (567h23m2.362129929s) [upload=499337]: Next returned nil segment Sep 22 05:31:27.009357354 (567h23m2.362142472s) [upload=499337]: Waiting for error group managing segments... Sep 22 05:31:27.018549517 (567h23m2.371334634s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.019540115 (567h23m2.372325234s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020276638 (567h23m2.373061756s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020786695 (567h23m2.373571803s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021133024 (567h23m2.373918133s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021696293 (567h23m2.37448141s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021726819 (567h23m2.374511927s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022237469 (567h23m2.375022586s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022647758 (567h23m2.375432876s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022916483 (567h23m2.3757016s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023451627 (567h23m2.376236745s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023639529 (567h23m2.376424648s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:30.479420128 (567h23m5.832205236s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; manager closed: conn closed: op Sep 22 05:31:56.821587941 (567h23m32.174373059s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821645730 (567h23m32.174430847s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821723946 (567h23m32.174509064s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; context deadline exceeded Sep 22 05:31:56.821747761 (567h23m32.174532879s) [seg_pos={0 0}, upload=499337]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:31:56.838365960 (567h23m32.191151059s) [upload=499337]: Done uploading segments... Sep 22 05:31:56.838377933 (567h23m32.191163031s) [upload=499337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 437.615009ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X...X.....X....XXX..........X.X.......XXX....X...........] duration: 613.480025ms FAIL (2025-09-22 05:48:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 613.724835ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.X..X..................X...............................] duration: 468.496702ms FAIL (2025-09-22 05:48:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment... Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment. Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments... Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments... Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 468.525054ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X...X.....X....XXX..........X.X.......XXX....X..........] duration: 410.31628ms FAIL (2025-09-22 05:48:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 410.332842ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......XXX.X.....X.......X.X...X..XX.........................] duration: 501.693115ms FAIL (2025-09-22 05:48:18 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:31:26.835749533 (567h23m2.188534662s) [upload=499337]: Starting upload Sep 22 05:31:26.835842508 (567h23m2.188627636s) [upload=499337]: Done starting upload Sep 22 05:31:26.836008360 (567h23m2.188793499s) [upload=499337]: Uploading segments... Sep 22 05:31:26.836536982 (567h23m2.18932211s) [upload=499337]: Got next segment. Inline: false Sep 22 05:31:26.836604177 (567h23m2.189389316s) [upload=499337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:31:26.836610950 (567h23m2.189396078s) [upload=499337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:31:26.856276449 (567h23m2.209061577s) [seg_pos={0 0}, upload=499337]: Begin upload segment... Sep 22 05:31:26.856314771 (567h23m2.209099899s) [seg_pos={0 0}, upload=499337]: Done begin upload segment. Sep 22 05:31:26.856372709 (567h23m2.209157838s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856423275 (567h23m2.209208413s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856888909 (567h23m2.209674037s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856930226 (567h23m2.209715354s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857336809 (567h23m2.210121948s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857426228 (567h23m2.210211355s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857842308 (567h23m2.210627436s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857850002 (567h23m2.210635131s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858275311 (567h23m2.211060449s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858418950 (567h23m2.211204089s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858708293 (567h23m2.211493421s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858912537 (567h23m2.211697665s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859330581 (567h23m2.212115709s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859801715 (567h23m2.212586844s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.860342481 (567h23m2.213127609s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.861314445 (567h23m2.214099573s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:27.009344801 (567h23m2.362129929s) [upload=499337]: Next returned nil segment Sep 22 05:31:27.009357354 (567h23m2.362142472s) [upload=499337]: Waiting for error group managing segments... Sep 22 05:31:27.018549517 (567h23m2.371334634s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.019540115 (567h23m2.372325234s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020276638 (567h23m2.373061756s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020786695 (567h23m2.373571803s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021133024 (567h23m2.373918133s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021696293 (567h23m2.37448141s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021726819 (567h23m2.374511927s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022237469 (567h23m2.375022586s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022647758 (567h23m2.375432876s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022916483 (567h23m2.3757016s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023451627 (567h23m2.376236745s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023639529 (567h23m2.376424648s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:30.479420128 (567h23m5.832205236s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; manager closed: conn closed: op Sep 22 05:31:56.821587941 (567h23m32.174373059s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821645730 (567h23m32.174430847s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821723946 (567h23m32.174509064s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; context deadline exceeded Sep 22 05:31:56.821747761 (567h23m32.174532879s) [seg_pos={0 0}, upload=499337]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:31:56.838365960 (567h23m32.191151059s) [upload=499337]: Done uploading segments... Sep 22 05:31:56.838377933 (567h23m32.191163031s) [upload=499337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 501.7165ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X...X.....X....XXX..........X.X.......XXX....X.........] duration: 402.067417ms FAIL (2025-09-22 05:47:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 402.083908ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......XXX.X.....X.......X.X...X..XX........................] duration: 828.733303ms FAIL (2025-09-22 05:47:29 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:31:26.835749533 (567h23m2.188534662s) [upload=499337]: Starting upload Sep 22 05:31:26.835842508 (567h23m2.188627636s) [upload=499337]: Done starting upload Sep 22 05:31:26.836008360 (567h23m2.188793499s) [upload=499337]: Uploading segments... Sep 22 05:31:26.836536982 (567h23m2.18932211s) [upload=499337]: Got next segment. Inline: false Sep 22 05:31:26.836604177 (567h23m2.189389316s) [upload=499337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:31:26.836610950 (567h23m2.189396078s) [upload=499337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:31:26.856276449 (567h23m2.209061577s) [seg_pos={0 0}, upload=499337]: Begin upload segment... Sep 22 05:31:26.856314771 (567h23m2.209099899s) [seg_pos={0 0}, upload=499337]: Done begin upload segment. Sep 22 05:31:26.856372709 (567h23m2.209157838s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856423275 (567h23m2.209208413s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856888909 (567h23m2.209674037s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856930226 (567h23m2.209715354s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857336809 (567h23m2.210121948s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857426228 (567h23m2.210211355s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857842308 (567h23m2.210627436s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857850002 (567h23m2.210635131s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858275311 (567h23m2.211060449s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858418950 (567h23m2.211204089s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858708293 (567h23m2.211493421s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858912537 (567h23m2.211697665s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859330581 (567h23m2.212115709s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859801715 (567h23m2.212586844s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.860342481 (567h23m2.213127609s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.861314445 (567h23m2.214099573s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:27.009344801 (567h23m2.362129929s) [upload=499337]: Next returned nil segment Sep 22 05:31:27.009357354 (567h23m2.362142472s) [upload=499337]: Waiting for error group managing segments... Sep 22 05:31:27.018549517 (567h23m2.371334634s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.019540115 (567h23m2.372325234s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020276638 (567h23m2.373061756s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020786695 (567h23m2.373571803s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021133024 (567h23m2.373918133s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021696293 (567h23m2.37448141s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021726819 (567h23m2.374511927s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022237469 (567h23m2.375022586s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022647758 (567h23m2.375432876s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022916483 (567h23m2.3757016s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023451627 (567h23m2.376236745s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023639529 (567h23m2.376424648s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:30.479420128 (567h23m5.832205236s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; manager closed: conn closed: op Sep 22 05:31:56.821587941 (567h23m32.174373059s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821645730 (567h23m32.174430847s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821723946 (567h23m32.174509064s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; context deadline exceeded Sep 22 05:31:56.821747761 (567h23m32.174532879s) [seg_pos={0 0}, upload=499337]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:31:56.838365960 (567h23m32.191151059s) [upload=499337]: Done uploading segments... Sep 22 05:31:56.838377933 (567h23m32.191163031s) [upload=499337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 828.752058ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.X..X..................X..............................] duration: 371.946616ms FAIL (2025-09-22 05:47:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment... Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment. Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments... Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments... Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 371.971915ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X...X.....X....XXX..........X.X.......XXX....X........] duration: 419.71554ms FAIL (2025-09-22 05:47:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 419.730108ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......X...X.....X....XXX..........X.X.......XXX....X.......] duration: 437.389821ms FAIL (2025-09-22 05:46:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 437.4418ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [........X...X.....X....XXX..........X.X.......XXX....X......] duration: 413.436235ms FAIL (2025-09-22 05:46:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 413.448898ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.X..X..................X.............................] duration: 797.767445ms FAIL (2025-09-22 05:46:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment... Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment. Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments... Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments... Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 797.788225ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.......XXX.X.....X.......X.X...X..XX.......................] duration: 568.068241ms FAIL (2025-09-22 05:46:15 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:31:26.835749533 (567h23m2.188534662s) [upload=499337]: Starting upload Sep 22 05:31:26.835842508 (567h23m2.188627636s) [upload=499337]: Done starting upload Sep 22 05:31:26.836008360 (567h23m2.188793499s) [upload=499337]: Uploading segments... Sep 22 05:31:26.836536982 (567h23m2.18932211s) [upload=499337]: Got next segment. Inline: false Sep 22 05:31:26.836604177 (567h23m2.189389316s) [upload=499337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:31:26.836610950 (567h23m2.189396078s) [upload=499337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:31:26.856276449 (567h23m2.209061577s) [seg_pos={0 0}, upload=499337]: Begin upload segment... Sep 22 05:31:26.856314771 (567h23m2.209099899s) [seg_pos={0 0}, upload=499337]: Done begin upload segment. Sep 22 05:31:26.856372709 (567h23m2.209157838s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856423275 (567h23m2.209208413s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856888909 (567h23m2.209674037s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856930226 (567h23m2.209715354s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857336809 (567h23m2.210121948s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857426228 (567h23m2.210211355s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857842308 (567h23m2.210627436s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857850002 (567h23m2.210635131s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858275311 (567h23m2.211060449s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858418950 (567h23m2.211204089s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858708293 (567h23m2.211493421s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858912537 (567h23m2.211697665s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859330581 (567h23m2.212115709s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859801715 (567h23m2.212586844s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.860342481 (567h23m2.213127609s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.861314445 (567h23m2.214099573s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:27.009344801 (567h23m2.362129929s) [upload=499337]: Next returned nil segment Sep 22 05:31:27.009357354 (567h23m2.362142472s) [upload=499337]: Waiting for error group managing segments... Sep 22 05:31:27.018549517 (567h23m2.371334634s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.019540115 (567h23m2.372325234s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020276638 (567h23m2.373061756s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020786695 (567h23m2.373571803s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021133024 (567h23m2.373918133s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021696293 (567h23m2.37448141s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021726819 (567h23m2.374511927s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022237469 (567h23m2.375022586s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022647758 (567h23m2.375432876s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022916483 (567h23m2.3757016s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023451627 (567h23m2.376236745s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023639529 (567h23m2.376424648s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:30.479420128 (567h23m5.832205236s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; manager closed: conn closed: op Sep 22 05:31:56.821587941 (567h23m32.174373059s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821645730 (567h23m32.174430847s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821723946 (567h23m32.174509064s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; context deadline exceeded Sep 22 05:31:56.821747761 (567h23m32.174532879s) [seg_pos={0 0}, upload=499337]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:31:56.838365960 (567h23m32.191151059s) [upload=499337]: Done uploading segments... Sep 22 05:31:56.838377933 (567h23m32.191163031s) [upload=499337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 568.09391ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.........X...X.....X....XXX..........X.X.......XXX....X.....] duration: 502.596503ms FAIL (2025-09-22 05:45:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 502.616509ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.......XXX.X.....X.......X.X...X..XX......................] duration: 434.471425ms FAIL (2025-09-22 05:45:51 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:31:26.835749533 (567h23m2.188534662s) [upload=499337]: Starting upload Sep 22 05:31:26.835842508 (567h23m2.188627636s) [upload=499337]: Done starting upload Sep 22 05:31:26.836008360 (567h23m2.188793499s) [upload=499337]: Uploading segments... Sep 22 05:31:26.836536982 (567h23m2.18932211s) [upload=499337]: Got next segment. Inline: false Sep 22 05:31:26.836604177 (567h23m2.189389316s) [upload=499337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:31:26.836610950 (567h23m2.189396078s) [upload=499337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:31:26.856276449 (567h23m2.209061577s) [seg_pos={0 0}, upload=499337]: Begin upload segment... Sep 22 05:31:26.856314771 (567h23m2.209099899s) [seg_pos={0 0}, upload=499337]: Done begin upload segment. Sep 22 05:31:26.856372709 (567h23m2.209157838s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856423275 (567h23m2.209208413s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856888909 (567h23m2.209674037s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856930226 (567h23m2.209715354s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857336809 (567h23m2.210121948s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857426228 (567h23m2.210211355s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857842308 (567h23m2.210627436s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857850002 (567h23m2.210635131s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858275311 (567h23m2.211060449s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858418950 (567h23m2.211204089s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858708293 (567h23m2.211493421s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858912537 (567h23m2.211697665s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859330581 (567h23m2.212115709s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859801715 (567h23m2.212586844s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.860342481 (567h23m2.213127609s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.861314445 (567h23m2.214099573s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:27.009344801 (567h23m2.362129929s) [upload=499337]: Next returned nil segment Sep 22 05:31:27.009357354 (567h23m2.362142472s) [upload=499337]: Waiting for error group managing segments... Sep 22 05:31:27.018549517 (567h23m2.371334634s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.019540115 (567h23m2.372325234s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020276638 (567h23m2.373061756s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020786695 (567h23m2.373571803s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021133024 (567h23m2.373918133s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021696293 (567h23m2.37448141s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021726819 (567h23m2.374511927s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022237469 (567h23m2.375022586s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022647758 (567h23m2.375432876s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022916483 (567h23m2.3757016s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023451627 (567h23m2.376236745s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023639529 (567h23m2.376424648s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:30.479420128 (567h23m5.832205236s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; manager closed: conn closed: op Sep 22 05:31:56.821587941 (567h23m32.174373059s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821645730 (567h23m32.174430847s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821723946 (567h23m32.174509064s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; context deadline exceeded Sep 22 05:31:56.821747761 (567h23m32.174532879s) [seg_pos={0 0}, upload=499337]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:31:56.838365960 (567h23m32.191151059s) [upload=499337]: Done uploading segments... Sep 22 05:31:56.838377933 (567h23m32.191163031s) [upload=499337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 434.494718ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.X..X..................X............................] duration: 354.194319ms FAIL (2025-09-22 05:45:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment... Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment. Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments... Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments... Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 354.213815ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.X.......XXX.X.....X.......X.X...X..XX.....................] duration: 419.143577ms FAIL (2025-09-22 05:45:33 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:31:26.835749533 (567h23m2.188534662s) [upload=499337]: Starting upload Sep 22 05:31:26.835842508 (567h23m2.188627636s) [upload=499337]: Done starting upload Sep 22 05:31:26.836008360 (567h23m2.188793499s) [upload=499337]: Uploading segments... Sep 22 05:31:26.836536982 (567h23m2.18932211s) [upload=499337]: Got next segment. Inline: false Sep 22 05:31:26.836604177 (567h23m2.189389316s) [upload=499337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:31:26.836610950 (567h23m2.189396078s) [upload=499337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:31:26.856276449 (567h23m2.209061577s) [seg_pos={0 0}, upload=499337]: Begin upload segment... Sep 22 05:31:26.856314771 (567h23m2.209099899s) [seg_pos={0 0}, upload=499337]: Done begin upload segment. Sep 22 05:31:26.856372709 (567h23m2.209157838s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856423275 (567h23m2.209208413s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856888909 (567h23m2.209674037s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.856930226 (567h23m2.209715354s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857336809 (567h23m2.210121948s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857426228 (567h23m2.210211355s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857842308 (567h23m2.210627436s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.857850002 (567h23m2.210635131s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858275311 (567h23m2.211060449s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858418950 (567h23m2.211204089s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858708293 (567h23m2.211493421s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.858912537 (567h23m2.211697665s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859330581 (567h23m2.212115709s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.859801715 (567h23m2.212586844s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.860342481 (567h23m2.213127609s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:26.861314445 (567h23m2.214099573s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Uploading piece... Sep 22 05:31:27.009344801 (567h23m2.362129929s) [upload=499337]: Next returned nil segment Sep 22 05:31:27.009357354 (567h23m2.362142472s) [upload=499337]: Waiting for error group managing segments... Sep 22 05:31:27.018549517 (567h23m2.371334634s) [noise=true , address=45.91.116.10:4083 , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.019540115 (567h23m2.372325234s) [noise=true , address=45.91.116.12:4226 , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020276638 (567h23m2.373061756s) [noise=true , address=77.90.31.15:4697 , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.020786695 (567h23m2.373571803s) [noise=true , address=77.90.31.12:4592 , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021133024 (567h23m2.373918133s) [noise=true , address=45.91.116.12:4227 , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021696293 (567h23m2.37448141s) [noise=true , address=77.90.31.21:4878 , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.021726819 (567h23m2.374511927s) [noise=true , address=45.91.116.5:4297 , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022237469 (567h23m2.375022586s) [noise=true , address=77.90.31.10:4520 , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022647758 (567h23m2.375432876s) [noise=true , address=45.91.116.6:4975 , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.022916483 (567h23m2.3757016s) [noise=true , address=77.90.31.16:4734 , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023451627 (567h23m2.376236745s) [noise=true , address=77.90.31.15:4698 , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:27.023639529 (567h23m2.376424648s) [noise=true , address=77.90.31.19:4841 , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil> Sep 22 05:31:30.479420128 (567h23m5.832205236s) [noise=true , address=145.79.227.16:4912 , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; manager closed: conn closed: op Sep 22 05:31:56.821587941 (567h23m32.174373059s) [noise=true , address=145.79.227.11:4731 , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821645730 (567h23m32.174430847s) [noise=true , address=145.79.227.11:4732 , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:31:56.821723946 (567h23m32.174509064s) [noise=true , address=145.79.227.5:4222 , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; context deadline exceeded Sep 22 05:31:56.821747761 (567h23m32.174532879s) [seg_pos={0 0}, upload=499337]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:31:56.838365960 (567h23m32.191151059s) [upload=499337]: Done uploading segments... Sep 22 05:31:56.838377933 (567h23m32.191163031s) [upload=499337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 419.169987ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.X..X..................X...........................] duration: 373.954536ms FAIL (2025-09-22 05:45:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment... Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment. Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments... Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments... Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 373.97237ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..........X...X.....X....XXX..........X.X.......XXX....X....] duration: 405.418517ms FAIL (2025-09-22 05:45:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 405.436932ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.X..X..................X..........................] duration: 458.876378ms FAIL (2025-09-22 05:45:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment... Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment. Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments... Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments... Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 458.903019ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)