Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 122863 reports (Page 896 of 2458)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip:
4 out of 60 is failed (threshold=3) [......................X.X..X..................X.............]
duration:
409.001482ms
FAIL (2025-09-22 05:33:46 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment...
Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment.
Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment
Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments...
Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op
Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op
Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments...
Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 409.024085ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
9 out of 60 is failed (threshold=3) [....X..................X...X.....X....XXX..........X.X......]
duration:
403.120424ms
FAIL (2025-09-22 05:33:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 403.135842ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
14 out of 60 is failed (threshold=3) [......X...X.......X.X.......XXX.X.....X.......X.X...X..XX...]
duration:
423.236851ms
FAIL (2025-09-22 05:33:22 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 05:31:26.835749533 (567h23m2.188534662s) [upload=499337]: Starting upload
Sep 22 05:31:26.835842508 (567h23m2.188627636s) [upload=499337]: Done starting upload
Sep 22 05:31:26.836008360 (567h23m2.188793499s) [upload=499337]: Uploading segments...
Sep 22 05:31:26.836536982 (567h23m2.18932211s) [upload=499337]: Got next segment. Inline: false
Sep 22 05:31:26.836604177 (567h23m2.189389316s) [upload=499337]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:31:26.836610950 (567h23m2.189396078s) [upload=499337]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:31:26.856276449 (567h23m2.209061577s) [seg_pos={0 0}, upload=499337]: Begin upload segment...
Sep 22 05:31:26.856314771 (567h23m2.209099899s) [seg_pos={0 0}, upload=499337]: Done begin upload segment.
Sep 22 05:31:26.856372709 (567h23m2.209157838s) [noise=true , address=145.79.227.5:4222    , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.856423275 (567h23m2.209208413s) [noise=true , address=77.90.31.16:4734     , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.856888909 (567h23m2.209674037s) [noise=true , address=145.79.227.11:4731   , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.856930226 (567h23m2.209715354s) [noise=true , address=77.90.31.10:4520     , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.857336809 (567h23m2.210121948s) [noise=true , address=77.90.31.21:4878     , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.857426228 (567h23m2.210211355s) [noise=true , address=45.91.116.6:4975     , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.857842308 (567h23m2.210627436s) [noise=true , address=77.90.31.19:4841     , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.857850002 (567h23m2.210635131s) [noise=true , address=45.91.116.12:4226    , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.858275311 (567h23m2.211060449s) [noise=true , address=45.91.116.5:4297     , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.858418950 (567h23m2.211204089s) [noise=true , address=77.90.31.12:4592     , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.858708293 (567h23m2.211493421s) [noise=true , address=77.90.31.15:4697     , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.858912537 (567h23m2.211697665s) [noise=true , address=77.90.31.15:4698     , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.859330581 (567h23m2.212115709s) [noise=true , address=45.91.116.12:4227    , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.859801715 (567h23m2.212586844s) [noise=true , address=45.91.116.10:4083    , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.860342481 (567h23m2.213127609s) [noise=true , address=145.79.227.16:4912   , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.861314445 (567h23m2.214099573s) [noise=true , address=145.79.227.11:4732   , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:27.009344801 (567h23m2.362129929s) [upload=499337]: Next returned nil segment
Sep 22 05:31:27.009357354 (567h23m2.362142472s) [upload=499337]: Waiting for error group managing segments...
Sep 22 05:31:27.018549517 (567h23m2.371334634s) [noise=true , address=45.91.116.10:4083    , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.019540115 (567h23m2.372325234s) [noise=true , address=45.91.116.12:4226    , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.020276638 (567h23m2.373061756s) [noise=true , address=77.90.31.15:4697     , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.020786695 (567h23m2.373571803s) [noise=true , address=77.90.31.12:4592     , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.021133024 (567h23m2.373918133s) [noise=true , address=45.91.116.12:4227    , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.021696293 (567h23m2.37448141s) [noise=true , address=77.90.31.21:4878     , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.021726819 (567h23m2.374511927s) [noise=true , address=45.91.116.5:4297     , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.022237469 (567h23m2.375022586s) [noise=true , address=77.90.31.10:4520     , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.022647758 (567h23m2.375432876s) [noise=true , address=45.91.116.6:4975     , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.022916483 (567h23m2.3757016s) [noise=true , address=77.90.31.16:4734     , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.023451627 (567h23m2.376236745s) [noise=true , address=77.90.31.15:4698     , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.023639529 (567h23m2.376424648s) [noise=true , address=77.90.31.19:4841     , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:30.479420128 (567h23m5.832205236s) [noise=true , address=145.79.227.16:4912   , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; manager closed: conn closed: op
Sep 22 05:31:56.821587941 (567h23m32.174373059s) [noise=true , address=145.79.227.11:4731   , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:31:56.821645730 (567h23m32.174430847s) [noise=true , address=145.79.227.11:4732   , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:31:56.821723946 (567h23m32.174509064s) [noise=true , address=145.79.227.5:4222    , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; context deadline exceeded
Sep 22 05:31:56.821747761 (567h23m32.174532879s) [seg_pos={0 0}, upload=499337]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 05:31:56.838365960 (567h23m32.191151059s) [upload=499337]: Done uploading segments...
Sep 22 05:31:56.838377933 (567h23m32.191163031s) [upload=499337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 423.260144ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................]
duration:
5.320525367s
FAIL (2025-09-22 05:33:14 UTC)
orbiter warning: operation: Commit
uplink: metaclient: context deadline exceeded
Duration: 5.320548843s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip:
4 out of 60 is failed (threshold=3) [.......................X.X..X..................X............]
duration:
346.284984ms
FAIL (2025-09-22 05:33:06 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment...
Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment.
Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment
Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments...
Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op
Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op
Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments...
Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 346.312687ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
9 out of 60 is failed (threshold=3) [.....X..................X...X.....X....XXX..........X.X.....]
duration:
421.46351ms
FAIL (2025-09-22 05:32:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 421.494729ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip:
4 out of 60 is failed (threshold=3) [........................X.X..X..................X...........]
duration:
437.823866ms
FAIL (2025-09-22 05:32:22 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment...
Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment.
Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment
Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments...
Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op
Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op
Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments...
Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 437.852659ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................]
duration:
5.767725804s
FAIL (2025-09-22 05:32:21 UTC)
orbiter warning: operation: Commit
uplink: metaclient: context deadline exceeded
Duration: 5.767747004s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
14 out of 60 is failed (threshold=3) [.......X...X.......X.X.......XXX.X.....X.......X.X...X..XX..]
duration:
747.773978ms
FAIL (2025-09-22 05:32:14 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 05:31:26.835749533 (567h23m2.188534662s) [upload=499337]: Starting upload
Sep 22 05:31:26.835842508 (567h23m2.188627636s) [upload=499337]: Done starting upload
Sep 22 05:31:26.836008360 (567h23m2.188793499s) [upload=499337]: Uploading segments...
Sep 22 05:31:26.836536982 (567h23m2.18932211s) [upload=499337]: Got next segment. Inline: false
Sep 22 05:31:26.836604177 (567h23m2.189389316s) [upload=499337]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:31:26.836610950 (567h23m2.189396078s) [upload=499337]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:31:26.856276449 (567h23m2.209061577s) [seg_pos={0 0}, upload=499337]: Begin upload segment...
Sep 22 05:31:26.856314771 (567h23m2.209099899s) [seg_pos={0 0}, upload=499337]: Done begin upload segment.
Sep 22 05:31:26.856372709 (567h23m2.209157838s) [noise=true , address=145.79.227.5:4222    , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.856423275 (567h23m2.209208413s) [noise=true , address=77.90.31.16:4734     , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.856888909 (567h23m2.209674037s) [noise=true , address=145.79.227.11:4731   , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.856930226 (567h23m2.209715354s) [noise=true , address=77.90.31.10:4520     , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.857336809 (567h23m2.210121948s) [noise=true , address=77.90.31.21:4878     , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.857426228 (567h23m2.210211355s) [noise=true , address=45.91.116.6:4975     , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.857842308 (567h23m2.210627436s) [noise=true , address=77.90.31.19:4841     , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.857850002 (567h23m2.210635131s) [noise=true , address=45.91.116.12:4226    , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.858275311 (567h23m2.211060449s) [noise=true , address=45.91.116.5:4297     , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.858418950 (567h23m2.211204089s) [noise=true , address=77.90.31.12:4592     , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.858708293 (567h23m2.211493421s) [noise=true , address=77.90.31.15:4697     , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.858912537 (567h23m2.211697665s) [noise=true , address=77.90.31.15:4698     , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.859330581 (567h23m2.212115709s) [noise=true , address=45.91.116.12:4227    , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.859801715 (567h23m2.212586844s) [noise=true , address=45.91.116.10:4083    , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.860342481 (567h23m2.213127609s) [noise=true , address=145.79.227.16:4912   , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.861314445 (567h23m2.214099573s) [noise=true , address=145.79.227.11:4732   , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:27.009344801 (567h23m2.362129929s) [upload=499337]: Next returned nil segment
Sep 22 05:31:27.009357354 (567h23m2.362142472s) [upload=499337]: Waiting for error group managing segments...
Sep 22 05:31:27.018549517 (567h23m2.371334634s) [noise=true , address=45.91.116.10:4083    , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.019540115 (567h23m2.372325234s) [noise=true , address=45.91.116.12:4226    , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.020276638 (567h23m2.373061756s) [noise=true , address=77.90.31.15:4697     , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.020786695 (567h23m2.373571803s) [noise=true , address=77.90.31.12:4592     , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.021133024 (567h23m2.373918133s) [noise=true , address=45.91.116.12:4227    , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.021696293 (567h23m2.37448141s) [noise=true , address=77.90.31.21:4878     , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.021726819 (567h23m2.374511927s) [noise=true , address=45.91.116.5:4297     , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.022237469 (567h23m2.375022586s) [noise=true , address=77.90.31.10:4520     , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.022647758 (567h23m2.375432876s) [noise=true , address=45.91.116.6:4975     , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.022916483 (567h23m2.3757016s) [noise=true , address=77.90.31.16:4734     , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.023451627 (567h23m2.376236745s) [noise=true , address=77.90.31.15:4698     , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.023639529 (567h23m2.376424648s) [noise=true , address=77.90.31.19:4841     , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:30.479420128 (567h23m5.832205236s) [noise=true , address=145.79.227.16:4912   , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; manager closed: conn closed: op
Sep 22 05:31:56.821587941 (567h23m32.174373059s) [noise=true , address=145.79.227.11:4731   , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:31:56.821645730 (567h23m32.174430847s) [noise=true , address=145.79.227.11:4732   , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:31:56.821723946 (567h23m32.174509064s) [noise=true , address=145.79.227.5:4222    , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; context deadline exceeded
Sep 22 05:31:56.821747761 (567h23m32.174532879s) [seg_pos={0 0}, upload=499337]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 05:31:56.838365960 (567h23m32.191151059s) [upload=499337]: Done uploading segments...
Sep 22 05:31:56.838377933 (567h23m32.191163031s) [upload=499337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 747.796389ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
9 out of 60 is failed (threshold=3) [......X..................X...X.....X....XXX..........X.X....]
duration:
428.022682ms
FAIL (2025-09-22 05:32:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 428.167944ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
14 out of 60 is failed (threshold=3) [........X...X.......X.X.......XXX.X.....X.......X.X...X..XX.]
duration:
392.475516ms
FAIL (2025-09-22 05:31:57 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 05:31:26.835749533 (567h23m2.188534662s) [upload=499337]: Starting upload
Sep 22 05:31:26.835842508 (567h23m2.188627636s) [upload=499337]: Done starting upload
Sep 22 05:31:26.836008360 (567h23m2.188793499s) [upload=499337]: Uploading segments...
Sep 22 05:31:26.836536982 (567h23m2.18932211s) [upload=499337]: Got next segment. Inline: false
Sep 22 05:31:26.836604177 (567h23m2.189389316s) [upload=499337]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:31:26.836610950 (567h23m2.189396078s) [upload=499337]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:31:26.856276449 (567h23m2.209061577s) [seg_pos={0 0}, upload=499337]: Begin upload segment...
Sep 22 05:31:26.856314771 (567h23m2.209099899s) [seg_pos={0 0}, upload=499337]: Done begin upload segment.
Sep 22 05:31:26.856372709 (567h23m2.209157838s) [noise=true , address=145.79.227.5:4222    , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.856423275 (567h23m2.209208413s) [noise=true , address=77.90.31.16:4734     , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.856888909 (567h23m2.209674037s) [noise=true , address=145.79.227.11:4731   , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.856930226 (567h23m2.209715354s) [noise=true , address=77.90.31.10:4520     , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.857336809 (567h23m2.210121948s) [noise=true , address=77.90.31.21:4878     , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.857426228 (567h23m2.210211355s) [noise=true , address=45.91.116.6:4975     , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.857842308 (567h23m2.210627436s) [noise=true , address=77.90.31.19:4841     , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.857850002 (567h23m2.210635131s) [noise=true , address=45.91.116.12:4226    , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.858275311 (567h23m2.211060449s) [noise=true , address=45.91.116.5:4297     , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.858418950 (567h23m2.211204089s) [noise=true , address=77.90.31.12:4592     , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.858708293 (567h23m2.211493421s) [noise=true , address=77.90.31.15:4697     , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.858912537 (567h23m2.211697665s) [noise=true , address=77.90.31.15:4698     , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.859330581 (567h23m2.212115709s) [noise=true , address=45.91.116.12:4227    , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.859801715 (567h23m2.212586844s) [noise=true , address=45.91.116.10:4083    , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.860342481 (567h23m2.213127609s) [noise=true , address=145.79.227.16:4912   , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.861314445 (567h23m2.214099573s) [noise=true , address=145.79.227.11:4732   , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:27.009344801 (567h23m2.362129929s) [upload=499337]: Next returned nil segment
Sep 22 05:31:27.009357354 (567h23m2.362142472s) [upload=499337]: Waiting for error group managing segments...
Sep 22 05:31:27.018549517 (567h23m2.371334634s) [noise=true , address=45.91.116.10:4083    , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.019540115 (567h23m2.372325234s) [noise=true , address=45.91.116.12:4226    , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.020276638 (567h23m2.373061756s) [noise=true , address=77.90.31.15:4697     , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.020786695 (567h23m2.373571803s) [noise=true , address=77.90.31.12:4592     , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.021133024 (567h23m2.373918133s) [noise=true , address=45.91.116.12:4227    , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.021696293 (567h23m2.37448141s) [noise=true , address=77.90.31.21:4878     , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.021726819 (567h23m2.374511927s) [noise=true , address=45.91.116.5:4297     , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.022237469 (567h23m2.375022586s) [noise=true , address=77.90.31.10:4520     , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.022647758 (567h23m2.375432876s) [noise=true , address=45.91.116.6:4975     , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.022916483 (567h23m2.3757016s) [noise=true , address=77.90.31.16:4734     , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.023451627 (567h23m2.376236745s) [noise=true , address=77.90.31.15:4698     , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.023639529 (567h23m2.376424648s) [noise=true , address=77.90.31.19:4841     , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:30.479420128 (567h23m5.832205236s) [noise=true , address=145.79.227.16:4912   , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; manager closed: conn closed: op
Sep 22 05:31:56.821587941 (567h23m32.174373059s) [noise=true , address=145.79.227.11:4731   , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:31:56.821645730 (567h23m32.174430847s) [noise=true , address=145.79.227.11:4732   , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:31:56.821723946 (567h23m32.174509064s) [noise=true , address=145.79.227.5:4222    , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; context deadline exceeded
Sep 22 05:31:56.821747761 (567h23m32.174532879s) [seg_pos={0 0}, upload=499337]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 05:31:56.838365960 (567h23m32.191151059s) [upload=499337]: Done uploading segments...
Sep 22 05:31:56.838377933 (567h23m32.191163031s) [upload=499337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 392.490243ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
14 out of 60 is failed (threshold=3) [.........X...X.......X.X.......XXX.X.....X.......X.X...X..XX]
duration:
30.018005238s
FAIL (2025-09-22 05:31:56 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 05:31:26.835749533 (567h23m2.188534662s) [upload=499337]: Starting upload
Sep 22 05:31:26.835842508 (567h23m2.188627636s) [upload=499337]: Done starting upload
Sep 22 05:31:26.836008360 (567h23m2.188793499s) [upload=499337]: Uploading segments...
Sep 22 05:31:26.836536982 (567h23m2.18932211s) [upload=499337]: Got next segment. Inline: false
Sep 22 05:31:26.836604177 (567h23m2.189389316s) [upload=499337]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:31:26.836610950 (567h23m2.189396078s) [upload=499337]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:31:26.856276449 (567h23m2.209061577s) [seg_pos={0 0}, upload=499337]: Begin upload segment...
Sep 22 05:31:26.856314771 (567h23m2.209099899s) [seg_pos={0 0}, upload=499337]: Done begin upload segment.
Sep 22 05:31:26.856372709 (567h23m2.209157838s) [noise=true , address=145.79.227.5:4222    , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.856423275 (567h23m2.209208413s) [noise=true , address=77.90.31.16:4734     , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.856888909 (567h23m2.209674037s) [noise=true , address=145.79.227.11:4731   , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.856930226 (567h23m2.209715354s) [noise=true , address=77.90.31.10:4520     , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.857336809 (567h23m2.210121948s) [noise=true , address=77.90.31.21:4878     , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.857426228 (567h23m2.210211355s) [noise=true , address=45.91.116.6:4975     , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.857842308 (567h23m2.210627436s) [noise=true , address=77.90.31.19:4841     , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.857850002 (567h23m2.210635131s) [noise=true , address=45.91.116.12:4226    , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.858275311 (567h23m2.211060449s) [noise=true , address=45.91.116.5:4297     , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.858418950 (567h23m2.211204089s) [noise=true , address=77.90.31.12:4592     , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.858708293 (567h23m2.211493421s) [noise=true , address=77.90.31.15:4697     , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.858912537 (567h23m2.211697665s) [noise=true , address=77.90.31.15:4698     , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.859330581 (567h23m2.212115709s) [noise=true , address=45.91.116.12:4227    , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.859801715 (567h23m2.212586844s) [noise=true , address=45.91.116.10:4083    , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.860342481 (567h23m2.213127609s) [noise=true , address=145.79.227.16:4912   , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:26.861314445 (567h23m2.214099573s) [noise=true , address=145.79.227.11:4732   , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Uploading piece...
Sep 22 05:31:27.009344801 (567h23m2.362129929s) [upload=499337]: Next returned nil segment
Sep 22 05:31:27.009357354 (567h23m2.362142472s) [upload=499337]: Waiting for error group managing segments...
Sep 22 05:31:27.018549517 (567h23m2.371334634s) [noise=true , address=45.91.116.10:4083    , piece_id=QEXMDWEXTNAK6G2D57HXRTL7IDRBW4YTJU5CRNKEBZCU2KB4KQYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.019540115 (567h23m2.372325234s) [noise=true , address=45.91.116.12:4226    , piece_id=LQBHGIFJU5G6X236IZDZ5HAAE72JQOHGJNMHBVLQMBOFPWJRA6YQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.020276638 (567h23m2.373061756s) [noise=true , address=77.90.31.15:4697     , piece_id=4UZLG7IG7LNI6IIEOQNFCPVDYUL5EBPHUUADBEL7HTIG65UUOGBA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.020786695 (567h23m2.373571803s) [noise=true , address=77.90.31.12:4592     , piece_id=QPHSGW6YISMAVOKK42UVIA34Y5XVG7VRWOH4YNS4J47MB6BRYQOA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.021133024 (567h23m2.373918133s) [noise=true , address=45.91.116.12:4227    , piece_id=H72VXS2S24APAIVE235B43FQ2QQOKZBALS7M5YOYGRZ46CBNODOQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.021696293 (567h23m2.37448141s) [noise=true , address=77.90.31.21:4878     , piece_id=RFAS3WXHSBDUZB4N65FPRCLQMTLLUERT5MIKDJESWZNAKSAJ5FQQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.021726819 (567h23m2.374511927s) [noise=true , address=45.91.116.5:4297     , piece_id=LLTELCMD7KAR5BWPFCW3GN5QT2EV7H55Y7H63ZPLCM22LEUPCAXA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.022237469 (567h23m2.375022586s) [noise=true , address=77.90.31.10:4520     , piece_id=6J2K4Z4HSVFUO5NLXECOJNGVZYKZFCKE75MUTFEVEGGT6OMHNRAA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.022647758 (567h23m2.375432876s) [noise=true , address=45.91.116.6:4975     , piece_id=G66KGEC4PS55COOC6DDB5WJK5BDE2KOLLAE5TEJPXOTYK7NDSLKA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.022916483 (567h23m2.3757016s) [noise=true , address=77.90.31.16:4734     , piece_id=XX7JTCK2IIK4WHGKTJRH6L3BKIAXX26Z3BOUP5Z27WZGY3GPO74Q, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.023451627 (567h23m2.376236745s) [noise=true , address=77.90.31.15:4698     , piece_id=7464MCTJYS4IYXQBIAGMPBL6UWXHF776SEAYKGLCDJOP55B5LOPA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:27.023639529 (567h23m2.376424648s) [noise=true , address=77.90.31.19:4841     , piece_id=II4IFUVQJ53X2LBYSMU5J67IT55XC2F5WWBDVGUHJUA3ZRC4WAYA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: <nil>
Sep 22 05:31:30.479420128 (567h23m5.832205236s) [noise=true , address=145.79.227.16:4912   , piece_id=ZK7F3H22XCV2SRCYR4CEGAOZU2JVYJLXXGKLNYUAVKDBZ72PDYUA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; manager closed: conn closed: op
Sep 22 05:31:56.821587941 (567h23m32.174373059s) [noise=true , address=145.79.227.11:4731   , piece_id=TIH6T4DZR3KFORJGSR4XUG236CFJPW4WUFKKCN76WWWFNTUHFIHA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:31:56.821645730 (567h23m32.174430847s) [noise=true , address=145.79.227.11:4732   , piece_id=CLCC77ZBAZTFJUPLFTNRLQPA3HWJLKC26K5BUARPFXFVIF7GXYIA, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:31:56.821723946 (567h23m32.174509064s) [noise=true , address=145.79.227.5:4222    , piece_id=DSO23N5SNXL44PITFQK5PJCGHQ5R2IZISJ3MAOCZNYYAA5OGNHTQ, seg_pos={0 0}, upload=499337]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; context deadline exceeded
Sep 22 05:31:56.821747761 (567h23m32.174532879s) [seg_pos={0 0}, upload=499337]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 05:31:56.838365960 (567h23m32.191151059s) [upload=499337]: Done uploading segments...
Sep 22 05:31:56.838377933 (567h23m32.191163031s) [upload=499337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.018019365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip:
4 out of 60 is failed (threshold=3) [.........................X.X..X..................X..........]
duration:
699.644817ms
FAIL (2025-09-22 05:31:44 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment...
Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment.
Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment
Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments...
Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op
Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op
Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments...
Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 699.660747ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
9 out of 60 is failed (threshold=3) [.......X..................X...X.....X....XXX..........X.X...]
duration:
495.785893ms
FAIL (2025-09-22 05:31:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 495.809478ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................]
duration:
6.565181514s
FAIL (2025-09-22 05:31:30 UTC)
orbiter warning: operation: Commit
uplink: metaclient: context deadline exceeded
Duration: 6.565207303s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................]
duration:
6.247516319s
FAIL (2025-09-22 05:31:08 UTC)
orbiter warning: operation: Commit
uplink: metaclient: context deadline exceeded
Duration: 6.247534102s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
13 out of 60 is failed (threshold=3) [..........X...X.......X.X.......XXX.X.....X.......X.X...X..X]
duration:
30.017108179s
FAIL (2025-09-22 05:31:03 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Sep 22 05:30:33.850433908 (567h22m9.203219007s) [upload=499325]: Starting upload
Sep 22 05:30:33.850483822 (567h22m9.20326892s) [upload=499325]: Done starting upload
Sep 22 05:30:33.850599449 (567h22m9.203384557s) [upload=499325]: Uploading segments...
Sep 22 05:30:33.850709466 (567h22m9.203494574s) [upload=499325]: Got next segment. Inline: false
Sep 22 05:30:33.850750694 (567h22m9.203535802s) [upload=499325]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:30:33.850754140 (567h22m9.203539248s) [upload=499325]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:30:33.879659534 (567h22m9.232444661s) [seg_pos={0 0}, upload=499325]: Begin upload segment...
Sep 22 05:30:33.879694459 (567h22m9.232479597s) [seg_pos={0 0}, upload=499325]: Done begin upload segment.
Sep 22 05:30:33.879728222 (567h22m9.23251335s) [noise=true , address=145.79.227.16:4912   , piece_id=MGVL44CMIKXYG7MEDFRGXMSXHDGWQJYQOLZQRNEUF2MQRUJP2XRA, seg_pos={0 0}, upload=499325]: Uploading piece...
Sep 22 05:30:33.879799446 (567h22m9.232584574s) [noise=true , address=145.79.227.2:4113    , piece_id=RPCCBRMGKLX6CEESUOCAQO5NPKG5VM2F3ZPEQCC7NMIFRZF7XGJQ, seg_pos={0 0}, upload=499325]: Uploading piece...
Sep 22 05:30:33.880264829 (567h22m9.233049966s) [noise=true , address=77.90.31.15:4698     , piece_id=WFUKQATWFJ36TTCZIFZACIRNFXWL7ZNQ4R46XIPTHYZKFUZGGYZA, seg_pos={0 0}, upload=499325]: Uploading piece...
Sep 22 05:30:33.880502445 (567h22m9.233287573s) [noise=true , address=77.90.31.11:4553     , piece_id=O6KTEPTKKUVDZ2IMW7VKYMMOH3CEBI3GNJU5UOWIXQALP5RSMURA, seg_pos={0 0}, upload=499325]: Uploading piece...
Sep 22 05:30:33.880812417 (567h22m9.233597554s) [noise=true , address=145.79.227.12:4765   , piece_id=ULGKIJVS6GJTOLOBC7ZTLLKGZT4RGYBVRQM4KNAOHGTHBZOAWYRA, seg_pos={0 0}, upload=499325]: Uploading piece...
Sep 22 05:30:33.881182861 (567h22m9.23396799s) [noise=true , address=145.79.227.5:4223    , piece_id=S3VUVNY3ZXBE5267PXLV2COPXAQP6AJGRSDFEWXWZDUV3YZBCHSQ, seg_pos={0 0}, upload=499325]: Uploading piece...
Sep 22 05:30:33.881299189 (567h22m9.234084328s) [noise=true , address=77.90.31.17:4770     , piece_id=ATKZGEFASCUM2P3PX77CVNG5BYT4ZVV7XIOFJFMIOM5I7NSFVJ7Q, seg_pos={0 0}, upload=499325]: Uploading piece...
Sep 22 05:30:33.882171937 (567h22m9.234957065s) [noise=true , address=145.79.227.3:4151    , piece_id=4VNBYJE2ZCWAYCUQ6JUGCAKBRW4L2HNIRTDLQ4C7UG56AXSUIM4Q, seg_pos={0 0}, upload=499325]: Uploading piece...
Sep 22 05:30:33.882211852 (567h22m9.234996991s) [noise=true , address=77.90.31.21:4877     , piece_id=ZNOPVYAKGSJRBI72MW6HZNQ7IIENPKZ6OXEEOEL6RDO7T5UIQCCA, seg_pos={0 0}, upload=499325]: Uploading piece...
Sep 22 05:30:33.882807380 (567h22m9.235592518s) [noise=true , address=77.90.31.17:4771     , piece_id=CPDO6DZYIMTNYL2QEA32N7LI3MX5Q4D4CNVHSJFXXPPAWTPVMNPQ, seg_pos={0 0}, upload=499325]: Uploading piece...
Sep 22 05:30:33.882974784 (567h22m9.235759912s) [noise=true , address=77.90.31.18:4807     , piece_id=SQ6DELEBXL6HITDT3E2NVESM42PKCBLFMORF6PFRFQ7XXESF26DA, seg_pos={0 0}, upload=499325]: Uploading piece...
Sep 22 05:30:33.883671352 (567h22m9.236456479s) [noise=true , address=45.91.116.14:4189    , piece_id=YHYZPV2DO7IB4I267YPKFWJ646A3LPDDW6EVW7JSJVCHJ7GJDRHA, seg_pos={0 0}, upload=499325]: Uploading piece...
Sep 22 05:30:33.884087672 (567h22m9.236872801s) [noise=true , address=45.91.116.9:4262     , piece_id=OEXAVEZDHSEPI34DOX7QE5OQYAJZGEVYMV6TYD2ZOQDWH2PQCGHA, seg_pos={0 0}, upload=499325]: Uploading piece...
Sep 22 05:30:33.884717324 (567h22m9.237502453s) [noise=true , address=145.79.227.12:4767   , piece_id=4IRSN5DXMRWOG5K2JB7QDWFRGZ42M4AHN5VVDAFB7ZXHUZSCQ2LA, seg_pos={0 0}, upload=499325]: Uploading piece...
Sep 22 05:30:33.885053776 (567h22m9.237838913s) [noise=true , address=77.90.31.18:4805     , piece_id=QUSAAZ6PUAGIRVPQB6U5L6EXS2FW2Q4TM5E7SPS5OSKX3GXLLO6Q, seg_pos={0 0}, upload=499325]: Uploading piece...
Sep 22 05:30:33.886135065 (567h22m9.238920194s) [noise=true , address=45.91.116.10:4083    , piece_id=2KK5S2Q25QZY3A7NAPQ5BZCWIIXBB45GSKRYVAEFR5Z6PHRYSDRA, seg_pos={0 0}, upload=499325]: Uploading piece...
Sep 22 05:30:34.008547462 (567h22m9.36133259s) [upload=499325]: Next returned nil segment
Sep 22 05:30:34.008562590 (567h22m9.361347718s) [upload=499325]: Waiting for error group managing segments...
Sep 22 05:30:34.018202731 (567h22m9.370987859s) [noise=true , address=77.90.31.18:4805     , piece_id=QUSAAZ6PUAGIRVPQB6U5L6EXS2FW2Q4TM5E7SPS5OSKX3GXLLO6Q, seg_pos={0 0}, upload=499325]: Done uploading piece. err: <nil>
Sep 22 05:30:34.018874912 (567h22m9.37166005s) [noise=true , address=77.90.31.15:4698     , piece_id=WFUKQATWFJ36TTCZIFZACIRNFXWL7ZNQ4R46XIPTHYZKFUZGGYZA, seg_pos={0 0}, upload=499325]: Done uploading piece. err: <nil>
Sep 22 05:30:34.019159527 (567h22m9.371944655s) [noise=true , address=45.91.116.9:4262     , piece_id=OEXAVEZDHSEPI34DOX7QE5OQYAJZGEVYMV6TYD2ZOQDWH2PQCGHA, seg_pos={0 0}, upload=499325]: Done uploading piece. err: <nil>
Sep 22 05:30:34.019567963 (567h22m9.372353091s) [noise=true , address=77.90.31.18:4807     , piece_id=SQ6DELEBXL6HITDT3E2NVESM42PKCBLFMORF6PFRFQ7XXESF26DA, seg_pos={0 0}, upload=499325]: Done uploading piece. err: <nil>
Sep 22 05:30:34.020243621 (567h22m9.373028749s) [noise=true , address=45.91.116.14:4189    , piece_id=YHYZPV2DO7IB4I267YPKFWJ646A3LPDDW6EVW7JSJVCHJ7GJDRHA, seg_pos={0 0}, upload=499325]: Done uploading piece. err: <nil>
Sep 22 05:30:34.020368815 (567h22m9.373153944s) [noise=true , address=77.90.31.11:4553     , piece_id=O6KTEPTKKUVDZ2IMW7VKYMMOH3CEBI3GNJU5UOWIXQALP5RSMURA, seg_pos={0 0}, upload=499325]: Done uploading piece. err: <nil>
Sep 22 05:30:34.020817137 (567h22m9.373602276s) [noise=true , address=77.90.31.17:4770     , piece_id=ATKZGEFASCUM2P3PX77CVNG5BYT4ZVV7XIOFJFMIOM5I7NSFVJ7Q, seg_pos={0 0}, upload=499325]: Done uploading piece. err: <nil>
Sep 22 05:30:34.021754787 (567h22m9.374539925s) [noise=true , address=77.90.31.21:4877     , piece_id=ZNOPVYAKGSJRBI72MW6HZNQ7IIENPKZ6OXEEOEL6RDO7T5UIQCCA, seg_pos={0 0}, upload=499325]: Done uploading piece. err: <nil>
Sep 22 05:30:34.022170247 (567h22m9.374955374s) [noise=true , address=77.90.31.17:4771     , piece_id=CPDO6DZYIMTNYL2QEA32N7LI3MX5Q4D4CNVHSJFXXPPAWTPVMNPQ, seg_pos={0 0}, upload=499325]: Done uploading piece. err: <nil>
Sep 22 05:30:34.022686826 (567h22m9.375471954s) [noise=true , address=45.91.116.10:4083    , piece_id=2KK5S2Q25QZY3A7NAPQ5BZCWIIXBB45GSKRYVAEFR5Z6PHRYSDRA, seg_pos={0 0}, upload=499325]: Done uploading piece. err: <nil>
Sep 22 05:30:37.842834797 (567h22m13.195619925s) [noise=true , address=145.79.227.5:4223    , piece_id=S3VUVNY3ZXBE5267PXLV2COPXAQP6AJGRSDFEWXWZDUV3YZBCHSQ, seg_pos={0 0}, upload=499325]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op
Sep 22 05:31:03.835626727 (567h22m39.188411854s) [noise=true , address=145.79.227.12:4767   , piece_id=4IRSN5DXMRWOG5K2JB7QDWFRGZ42M4AHN5VVDAFB7ZXHUZSCQ2LA, seg_pos={0 0}, upload=499325]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:31:03.835711105 (567h22m39.188496233s) [noise=true , address=145.79.227.16:4912   , piece_id=MGVL44CMIKXYG7MEDFRGXMSXHDGWQJYQOLZQRNEUF2MQRUJP2XRA, seg_pos={0 0}, upload=499325]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:31:03.835772339 (567h22m39.188557468s) [noise=true , address=145.79.227.2:4113    , piece_id=RPCCBRMGKLX6CEESUOCAQO5NPKG5VM2F3ZPEQCC7NMIFRZF7XGJQ, seg_pos={0 0}, upload=499325]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: rpc: dial tcp 145.79.227.2:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:31:03.835855235 (567h22m39.188640363s) [noise=true , address=145.79.227.3:4151    , piece_id=4VNBYJE2ZCWAYCUQ6JUGCAKBRW4L2HNIRTDLQ4C7UG56AXSUIM4Q, seg_pos={0 0}, upload=499325]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: context deadline exceeded; context deadline exceeded
Sep 22 05:31:03.835982965 (567h22m39.188768103s) [noise=true , address=145.79.227.12:4765   , piece_id=ULGKIJVS6GJTOLOBC7ZTLLKGZT4RGYBVRQM4KNAOHGTHBZOAWYRA, seg_pos={0 0}, upload=499325]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:31:03.836013662 (567h22m39.18879879s) [seg_pos={0 0}, upload=499325]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Sep 22 05:31:03.850379381 (567h22m39.203164509s) [upload=499325]: Done uploading segments...
Sep 22 05:31:03.850397225 (567h22m39.203182353s) [upload=499325]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.017128688s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip:
4 out of 60 is failed (threshold=3) [..........................X.X..X..................X.........]
duration:
433.489859ms
FAIL (2025-09-22 05:30:38 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment...
Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment.
Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment
Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments...
Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op
Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op
Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments...
Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 433.512642ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
9 out of 60 is failed (threshold=3) [........X..................X...X.....X....XXX..........X.X..]
duration:
572.74939ms
FAIL (2025-09-22 05:30:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 572.764849ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................]
duration:
6.926707616s
FAIL (2025-09-22 05:30:10 UTC)
orbiter warning: operation: Commit
uplink: metaclient: context deadline exceeded
Duration: 6.926733385s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
10 out of 60 is failed (threshold=3) [X........X..................X...X.....X....XXX..........X.X.]
duration:
672.906154ms
FAIL (2025-09-22 05:30:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 672.928155ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
10 out of 60 is failed (threshold=3) [.X........X..................X...X.....X....XXX..........X.X]
duration:
30.000836203s
FAIL (2025-09-22 05:30:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00085643s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
12 out of 60 is failed (threshold=3) [...........X...X.......X.X.......XXX.X.....X.......X.X...X..]
duration:
529.598265ms
FAIL (2025-09-22 05:30:02 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 05:28:29.803229127 (567h20m5.156014225s) [upload=499296]: Starting upload
Sep 22 05:28:29.803276756 (567h20m5.156061864s) [upload=499296]: Done starting upload
Sep 22 05:28:29.803344343 (567h20m5.156129441s) [upload=499296]: Uploading segments...
Sep 22 05:28:29.803556893 (567h20m5.156342001s) [upload=499296]: Got next segment. Inline: false
Sep 22 05:28:29.803580196 (567h20m5.156365304s) [upload=499296]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:28:29.803583984 (567h20m5.156369081s) [upload=499296]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:28:29.825189989 (567h20m5.177975087s) [seg_pos={0 0}, upload=499296]: Begin upload segment...
Sep 22 05:28:29.825210747 (567h20m5.177995846s) [seg_pos={0 0}, upload=499296]: Done begin upload segment.
Sep 22 05:28:29.825226928 (567h20m5.178012036s) [noise=true , address=45.91.116.11:4117    , piece_id=C2MIQBV4JFBRAD3DOTOB775NZEMD7YVAHJ5FHRDNDBZUT7FNT6JA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.825257595 (567h20m5.178042703s) [noise=true , address=145.79.227.6:4258    , piece_id=KZKJ3EJGNRCDYYKILOB3R4NTQKAK44M3Y4Z6Z67B4FG6XT62P6CQ, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.825511141 (567h20m5.178296239s) [noise=true , address=45.91.116.14:4191    , piece_id=O74UZOE7U5KFCFU5XQHGVMMWN65CV6WOFHWOLTIDJLFJHU6XSVFA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.825545035 (567h20m5.178330132s) [noise=true , address=77.90.31.17:4770     , piece_id=I4JB5UT3GPD7YLVVV74OJEY62IZUSJ3GRC7ERV47BNBDOOGXWFCA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.825728519 (567h20m5.178513627s) [noise=true , address=45.91.116.14:4189    , piece_id=MX2GWMKTZNUVCI4VH2V7GQJKHKX6WQFGILSH2CJA6WB6OILIEERA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.825775437 (567h20m5.178560545s) [noise=true , address=77.90.31.10:4520     , piece_id=SLTEJQGU4MQFIGWCS4EPL3BRSILVS5JF57FJ2I46WU5OAPTN5FHA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826041206 (567h20m5.178826314s) [noise=true , address=145.79.227.13:4802   , piece_id=5YWIAJRNNJRRTFUNL5PNEGWGFJBFEDK3L7SK425BBTSO7BF6LXRA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826052588 (567h20m5.178837696s) [noise=true , address=77.90.31.12:4592     , piece_id=5M4OPGMTWMWPK76RI62BYFIVOD4UFSLHVC5IA4JDGIPHUDJYT43Q, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826274654 (567h20m5.179059762s) [noise=true , address=145.79.227.14:4837   , piece_id=P2STYEHPN3EYE6HT3KTO3BYTJDLF52CIP6B42PRPIX2OBADXIJ2A, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826279944 (567h20m5.179065042s) [noise=true , address=45.91.116.10:4083    , piece_id=WKRO6QK7RN6PCDL4KXFNXQVCW7EYSPJGIDS5AGHC363T26ACXU6Q, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826489938 (567h20m5.179275046s) [noise=true , address=145.79.227.19:4078   , piece_id=NS2NAUDD4RQV4TCUAPSPBLIMITSIAX5KPWZQ5HXZVAKYRBAZWUVA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826513462 (567h20m5.17929857s) [noise=true , address=45.91.116.6:4975     , piece_id=XZPUJCBJHQKMFMRC5K2EC2VFC2GBKUWP4NJ5BHXTC62WEUBQXN7Q, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826747341 (567h20m5.179532449s) [noise=true , address=45.91.116.12:4227    , piece_id=H7UHE6ZHCOFBUFDSH4GMB3LQURR5EXM3J2LPN2763HQ7DHT2Z72Q, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826968717 (567h20m5.179753825s) [noise=true , address=77.90.31.19:4843     , piece_id=AHPHZA7SCLQYCESPMFATJYWQCZSU6X5YBDL65TNPNPD63OVUEOZQ, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.827201172 (567h20m5.17998628s) [noise=true , address=45.91.116.13:4154    , piece_id=ZICZ3XA7KXSY7TNK53HAH3WOJ7VCQ33UTQTPTOKHS5DWRLENC3XQ, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.827855411 (567h20m5.180640519s) [noise=true , address=45.91.116.8:4045     , piece_id=GKLKTCUBM22LS6W2ZMKI3GRJHSGW4XJVJZMMQH7PCSM7BXGWTUYQ, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.897638554 (567h20m5.250423662s) [upload=499296]: Next returned nil segment
Sep 22 05:28:29.897648964 (567h20m5.250434072s) [upload=499296]: Waiting for error group managing segments...
Sep 22 05:28:29.907305565 (567h20m5.260090673s) [noise=true , address=45.91.116.13:4154    , piece_id=ZICZ3XA7KXSY7TNK53HAH3WOJ7VCQ33UTQTPTOKHS5DWRLENC3XQ, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.907647867 (567h20m5.260432975s) [noise=true , address=45.91.116.11:4117    , piece_id=C2MIQBV4JFBRAD3DOTOB775NZEMD7YVAHJ5FHRDNDBZUT7FNT6JA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.908004927 (567h20m5.260790036s) [noise=true , address=77.90.31.12:4592     , piece_id=5M4OPGMTWMWPK76RI62BYFIVOD4UFSLHVC5IA4JDGIPHUDJYT43Q, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.908367628 (567h20m5.261152726s) [noise=true , address=45.91.116.14:4191    , piece_id=O74UZOE7U5KFCFU5XQHGVMMWN65CV6WOFHWOLTIDJLFJHU6XSVFA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.908711954 (567h20m5.261497062s) [noise=true , address=45.91.116.6:4975     , piece_id=XZPUJCBJHQKMFMRC5K2EC2VFC2GBKUWP4NJ5BHXTC62WEUBQXN7Q, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.909094382 (567h20m5.26187948s) [noise=true , address=45.91.116.12:4227    , piece_id=H7UHE6ZHCOFBUFDSH4GMB3LQURR5EXM3J2LPN2763HQ7DHT2Z72Q, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.909396058 (567h20m5.262181156s) [noise=true , address=45.91.116.8:4045     , piece_id=GKLKTCUBM22LS6W2ZMKI3GRJHSGW4XJVJZMMQH7PCSM7BXGWTUYQ, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.909938486 (567h20m5.262723584s) [noise=true , address=77.90.31.19:4843     , piece_id=AHPHZA7SCLQYCESPMFATJYWQCZSU6X5YBDL65TNPNPD63OVUEOZQ, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.910238909 (567h20m5.263024018s) [noise=true , address=77.90.31.17:4770     , piece_id=I4JB5UT3GPD7YLVVV74OJEY62IZUSJ3GRC7ERV47BNBDOOGXWFCA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.910541658 (567h20m5.263326755s) [noise=true , address=45.91.116.14:4189    , piece_id=MX2GWMKTZNUVCI4VH2V7GQJKHKX6WQFGILSH2CJA6WB6OILIEERA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.910834568 (567h20m5.263619665s) [noise=true , address=77.90.31.10:4520     , piece_id=SLTEJQGU4MQFIGWCS4EPL3BRSILVS5JF57FJ2I46WU5OAPTN5FHA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.911189984 (567h20m5.263975092s) [noise=true , address=45.91.116.10:4083    , piece_id=WKRO6QK7RN6PCDL4KXFNXQVCW7EYSPJGIDS5AGHC363T26ACXU6Q, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:32.076989427 (567h20m7.429774525s) [noise=true , address=145.79.227.14:4837   , piece_id=P2STYEHPN3EYE6HT3KTO3BYTJDLF52CIP6B42PRPIX2OBADXIJ2A, seg_pos={0 0}, upload=499296]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Sep 22 05:28:59.787412010 (567h20m35.140197139s) [noise=true , address=145.79.227.6:4258    , piece_id=KZKJ3EJGNRCDYYKILOB3R4NTQKAK44M3Y4Z6Z67B4FG6XT62P6CQ, seg_pos={0 0}, upload=499296]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:28:59.787437068 (567h20m35.140222206s) [noise=true , address=145.79.227.13:4802   , piece_id=5YWIAJRNNJRRTFUNL5PNEGWGFJBFEDK3L7SK425BBTSO7BF6LXRA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: context deadline exceeded; context deadline exceeded
Sep 22 05:28:59.787483675 (567h20m35.140268813s) [noise=true , address=145.79.227.19:4078   , piece_id=NS2NAUDD4RQV4TCUAPSPBLIMITSIAX5KPWZQ5HXZVAKYRBAZWUVA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:28:59.787512890 (567h20m35.140298028s) [seg_pos={0 0}, upload=499296]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 05:28:59.801215935 (567h20m35.154001063s) [upload=499296]: Done uploading segments...
Sep 22 05:28:59.801230542 (567h20m35.154015671s) [upload=499296]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 529.623252ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip:
4 out of 60 is failed (threshold=3) [...........................X.X..X..................X........]
duration:
393.787424ms
FAIL (2025-09-22 05:29:47 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment...
Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment.
Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment
Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments...
Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op
Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op
Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments...
Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 393.815407ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
12 out of 60 is failed (threshold=3) [............X...X.......X.X.......XXX.X.....X.......X.X...X.]
duration:
412.154953ms
FAIL (2025-09-22 05:29:20 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 05:28:29.803229127 (567h20m5.156014225s) [upload=499296]: Starting upload
Sep 22 05:28:29.803276756 (567h20m5.156061864s) [upload=499296]: Done starting upload
Sep 22 05:28:29.803344343 (567h20m5.156129441s) [upload=499296]: Uploading segments...
Sep 22 05:28:29.803556893 (567h20m5.156342001s) [upload=499296]: Got next segment. Inline: false
Sep 22 05:28:29.803580196 (567h20m5.156365304s) [upload=499296]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:28:29.803583984 (567h20m5.156369081s) [upload=499296]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:28:29.825189989 (567h20m5.177975087s) [seg_pos={0 0}, upload=499296]: Begin upload segment...
Sep 22 05:28:29.825210747 (567h20m5.177995846s) [seg_pos={0 0}, upload=499296]: Done begin upload segment.
Sep 22 05:28:29.825226928 (567h20m5.178012036s) [noise=true , address=45.91.116.11:4117    , piece_id=C2MIQBV4JFBRAD3DOTOB775NZEMD7YVAHJ5FHRDNDBZUT7FNT6JA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.825257595 (567h20m5.178042703s) [noise=true , address=145.79.227.6:4258    , piece_id=KZKJ3EJGNRCDYYKILOB3R4NTQKAK44M3Y4Z6Z67B4FG6XT62P6CQ, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.825511141 (567h20m5.178296239s) [noise=true , address=45.91.116.14:4191    , piece_id=O74UZOE7U5KFCFU5XQHGVMMWN65CV6WOFHWOLTIDJLFJHU6XSVFA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.825545035 (567h20m5.178330132s) [noise=true , address=77.90.31.17:4770     , piece_id=I4JB5UT3GPD7YLVVV74OJEY62IZUSJ3GRC7ERV47BNBDOOGXWFCA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.825728519 (567h20m5.178513627s) [noise=true , address=45.91.116.14:4189    , piece_id=MX2GWMKTZNUVCI4VH2V7GQJKHKX6WQFGILSH2CJA6WB6OILIEERA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.825775437 (567h20m5.178560545s) [noise=true , address=77.90.31.10:4520     , piece_id=SLTEJQGU4MQFIGWCS4EPL3BRSILVS5JF57FJ2I46WU5OAPTN5FHA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826041206 (567h20m5.178826314s) [noise=true , address=145.79.227.13:4802   , piece_id=5YWIAJRNNJRRTFUNL5PNEGWGFJBFEDK3L7SK425BBTSO7BF6LXRA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826052588 (567h20m5.178837696s) [noise=true , address=77.90.31.12:4592     , piece_id=5M4OPGMTWMWPK76RI62BYFIVOD4UFSLHVC5IA4JDGIPHUDJYT43Q, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826274654 (567h20m5.179059762s) [noise=true , address=145.79.227.14:4837   , piece_id=P2STYEHPN3EYE6HT3KTO3BYTJDLF52CIP6B42PRPIX2OBADXIJ2A, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826279944 (567h20m5.179065042s) [noise=true , address=45.91.116.10:4083    , piece_id=WKRO6QK7RN6PCDL4KXFNXQVCW7EYSPJGIDS5AGHC363T26ACXU6Q, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826489938 (567h20m5.179275046s) [noise=true , address=145.79.227.19:4078   , piece_id=NS2NAUDD4RQV4TCUAPSPBLIMITSIAX5KPWZQ5HXZVAKYRBAZWUVA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826513462 (567h20m5.17929857s) [noise=true , address=45.91.116.6:4975     , piece_id=XZPUJCBJHQKMFMRC5K2EC2VFC2GBKUWP4NJ5BHXTC62WEUBQXN7Q, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826747341 (567h20m5.179532449s) [noise=true , address=45.91.116.12:4227    , piece_id=H7UHE6ZHCOFBUFDSH4GMB3LQURR5EXM3J2LPN2763HQ7DHT2Z72Q, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826968717 (567h20m5.179753825s) [noise=true , address=77.90.31.19:4843     , piece_id=AHPHZA7SCLQYCESPMFATJYWQCZSU6X5YBDL65TNPNPD63OVUEOZQ, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.827201172 (567h20m5.17998628s) [noise=true , address=45.91.116.13:4154    , piece_id=ZICZ3XA7KXSY7TNK53HAH3WOJ7VCQ33UTQTPTOKHS5DWRLENC3XQ, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.827855411 (567h20m5.180640519s) [noise=true , address=45.91.116.8:4045     , piece_id=GKLKTCUBM22LS6W2ZMKI3GRJHSGW4XJVJZMMQH7PCSM7BXGWTUYQ, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.897638554 (567h20m5.250423662s) [upload=499296]: Next returned nil segment
Sep 22 05:28:29.897648964 (567h20m5.250434072s) [upload=499296]: Waiting for error group managing segments...
Sep 22 05:28:29.907305565 (567h20m5.260090673s) [noise=true , address=45.91.116.13:4154    , piece_id=ZICZ3XA7KXSY7TNK53HAH3WOJ7VCQ33UTQTPTOKHS5DWRLENC3XQ, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.907647867 (567h20m5.260432975s) [noise=true , address=45.91.116.11:4117    , piece_id=C2MIQBV4JFBRAD3DOTOB775NZEMD7YVAHJ5FHRDNDBZUT7FNT6JA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.908004927 (567h20m5.260790036s) [noise=true , address=77.90.31.12:4592     , piece_id=5M4OPGMTWMWPK76RI62BYFIVOD4UFSLHVC5IA4JDGIPHUDJYT43Q, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.908367628 (567h20m5.261152726s) [noise=true , address=45.91.116.14:4191    , piece_id=O74UZOE7U5KFCFU5XQHGVMMWN65CV6WOFHWOLTIDJLFJHU6XSVFA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.908711954 (567h20m5.261497062s) [noise=true , address=45.91.116.6:4975     , piece_id=XZPUJCBJHQKMFMRC5K2EC2VFC2GBKUWP4NJ5BHXTC62WEUBQXN7Q, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.909094382 (567h20m5.26187948s) [noise=true , address=45.91.116.12:4227    , piece_id=H7UHE6ZHCOFBUFDSH4GMB3LQURR5EXM3J2LPN2763HQ7DHT2Z72Q, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.909396058 (567h20m5.262181156s) [noise=true , address=45.91.116.8:4045     , piece_id=GKLKTCUBM22LS6W2ZMKI3GRJHSGW4XJVJZMMQH7PCSM7BXGWTUYQ, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.909938486 (567h20m5.262723584s) [noise=true , address=77.90.31.19:4843     , piece_id=AHPHZA7SCLQYCESPMFATJYWQCZSU6X5YBDL65TNPNPD63OVUEOZQ, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.910238909 (567h20m5.263024018s) [noise=true , address=77.90.31.17:4770     , piece_id=I4JB5UT3GPD7YLVVV74OJEY62IZUSJ3GRC7ERV47BNBDOOGXWFCA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.910541658 (567h20m5.263326755s) [noise=true , address=45.91.116.14:4189    , piece_id=MX2GWMKTZNUVCI4VH2V7GQJKHKX6WQFGILSH2CJA6WB6OILIEERA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.910834568 (567h20m5.263619665s) [noise=true , address=77.90.31.10:4520     , piece_id=SLTEJQGU4MQFIGWCS4EPL3BRSILVS5JF57FJ2I46WU5OAPTN5FHA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.911189984 (567h20m5.263975092s) [noise=true , address=45.91.116.10:4083    , piece_id=WKRO6QK7RN6PCDL4KXFNXQVCW7EYSPJGIDS5AGHC363T26ACXU6Q, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:32.076989427 (567h20m7.429774525s) [noise=true , address=145.79.227.14:4837   , piece_id=P2STYEHPN3EYE6HT3KTO3BYTJDLF52CIP6B42PRPIX2OBADXIJ2A, seg_pos={0 0}, upload=499296]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Sep 22 05:28:59.787412010 (567h20m35.140197139s) [noise=true , address=145.79.227.6:4258    , piece_id=KZKJ3EJGNRCDYYKILOB3R4NTQKAK44M3Y4Z6Z67B4FG6XT62P6CQ, seg_pos={0 0}, upload=499296]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:28:59.787437068 (567h20m35.140222206s) [noise=true , address=145.79.227.13:4802   , piece_id=5YWIAJRNNJRRTFUNL5PNEGWGFJBFEDK3L7SK425BBTSO7BF6LXRA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: context deadline exceeded; context deadline exceeded
Sep 22 05:28:59.787483675 (567h20m35.140268813s) [noise=true , address=145.79.227.19:4078   , piece_id=NS2NAUDD4RQV4TCUAPSPBLIMITSIAX5KPWZQ5HXZVAKYRBAZWUVA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:28:59.787512890 (567h20m35.140298028s) [seg_pos={0 0}, upload=499296]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 05:28:59.801215935 (567h20m35.154001063s) [upload=499296]: Done uploading segments...
Sep 22 05:28:59.801230542 (567h20m35.154015671s) [upload=499296]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 412.177033ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................]
duration:
5.351088546s
FAIL (2025-09-22 05:29:16 UTC)
orbiter warning: operation: Commit
uplink: metaclient: context deadline exceeded
Duration: 5.351107551s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip:
4 out of 60 is failed (threshold=3) [............................X.X..X..................X.......]
duration:
550.529495ms
FAIL (2025-09-22 05:29:03 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment...
Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment.
Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment
Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments...
Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op
Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op
Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments...
Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 550.55369ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
12 out of 60 is failed (threshold=3) [.............X...X.......X.X.......XXX.X.....X.......X.X...X]
duration:
30.016010751s
FAIL (2025-09-22 05:28:59 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 05:28:29.803229127 (567h20m5.156014225s) [upload=499296]: Starting upload
Sep 22 05:28:29.803276756 (567h20m5.156061864s) [upload=499296]: Done starting upload
Sep 22 05:28:29.803344343 (567h20m5.156129441s) [upload=499296]: Uploading segments...
Sep 22 05:28:29.803556893 (567h20m5.156342001s) [upload=499296]: Got next segment. Inline: false
Sep 22 05:28:29.803580196 (567h20m5.156365304s) [upload=499296]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:28:29.803583984 (567h20m5.156369081s) [upload=499296]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:28:29.825189989 (567h20m5.177975087s) [seg_pos={0 0}, upload=499296]: Begin upload segment...
Sep 22 05:28:29.825210747 (567h20m5.177995846s) [seg_pos={0 0}, upload=499296]: Done begin upload segment.
Sep 22 05:28:29.825226928 (567h20m5.178012036s) [noise=true , address=45.91.116.11:4117    , piece_id=C2MIQBV4JFBRAD3DOTOB775NZEMD7YVAHJ5FHRDNDBZUT7FNT6JA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.825257595 (567h20m5.178042703s) [noise=true , address=145.79.227.6:4258    , piece_id=KZKJ3EJGNRCDYYKILOB3R4NTQKAK44M3Y4Z6Z67B4FG6XT62P6CQ, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.825511141 (567h20m5.178296239s) [noise=true , address=45.91.116.14:4191    , piece_id=O74UZOE7U5KFCFU5XQHGVMMWN65CV6WOFHWOLTIDJLFJHU6XSVFA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.825545035 (567h20m5.178330132s) [noise=true , address=77.90.31.17:4770     , piece_id=I4JB5UT3GPD7YLVVV74OJEY62IZUSJ3GRC7ERV47BNBDOOGXWFCA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.825728519 (567h20m5.178513627s) [noise=true , address=45.91.116.14:4189    , piece_id=MX2GWMKTZNUVCI4VH2V7GQJKHKX6WQFGILSH2CJA6WB6OILIEERA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.825775437 (567h20m5.178560545s) [noise=true , address=77.90.31.10:4520     , piece_id=SLTEJQGU4MQFIGWCS4EPL3BRSILVS5JF57FJ2I46WU5OAPTN5FHA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826041206 (567h20m5.178826314s) [noise=true , address=145.79.227.13:4802   , piece_id=5YWIAJRNNJRRTFUNL5PNEGWGFJBFEDK3L7SK425BBTSO7BF6LXRA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826052588 (567h20m5.178837696s) [noise=true , address=77.90.31.12:4592     , piece_id=5M4OPGMTWMWPK76RI62BYFIVOD4UFSLHVC5IA4JDGIPHUDJYT43Q, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826274654 (567h20m5.179059762s) [noise=true , address=145.79.227.14:4837   , piece_id=P2STYEHPN3EYE6HT3KTO3BYTJDLF52CIP6B42PRPIX2OBADXIJ2A, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826279944 (567h20m5.179065042s) [noise=true , address=45.91.116.10:4083    , piece_id=WKRO6QK7RN6PCDL4KXFNXQVCW7EYSPJGIDS5AGHC363T26ACXU6Q, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826489938 (567h20m5.179275046s) [noise=true , address=145.79.227.19:4078   , piece_id=NS2NAUDD4RQV4TCUAPSPBLIMITSIAX5KPWZQ5HXZVAKYRBAZWUVA, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826513462 (567h20m5.17929857s) [noise=true , address=45.91.116.6:4975     , piece_id=XZPUJCBJHQKMFMRC5K2EC2VFC2GBKUWP4NJ5BHXTC62WEUBQXN7Q, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826747341 (567h20m5.179532449s) [noise=true , address=45.91.116.12:4227    , piece_id=H7UHE6ZHCOFBUFDSH4GMB3LQURR5EXM3J2LPN2763HQ7DHT2Z72Q, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.826968717 (567h20m5.179753825s) [noise=true , address=77.90.31.19:4843     , piece_id=AHPHZA7SCLQYCESPMFATJYWQCZSU6X5YBDL65TNPNPD63OVUEOZQ, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.827201172 (567h20m5.17998628s) [noise=true , address=45.91.116.13:4154    , piece_id=ZICZ3XA7KXSY7TNK53HAH3WOJ7VCQ33UTQTPTOKHS5DWRLENC3XQ, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.827855411 (567h20m5.180640519s) [noise=true , address=45.91.116.8:4045     , piece_id=GKLKTCUBM22LS6W2ZMKI3GRJHSGW4XJVJZMMQH7PCSM7BXGWTUYQ, seg_pos={0 0}, upload=499296]: Uploading piece...
Sep 22 05:28:29.897638554 (567h20m5.250423662s) [upload=499296]: Next returned nil segment
Sep 22 05:28:29.897648964 (567h20m5.250434072s) [upload=499296]: Waiting for error group managing segments...
Sep 22 05:28:29.907305565 (567h20m5.260090673s) [noise=true , address=45.91.116.13:4154    , piece_id=ZICZ3XA7KXSY7TNK53HAH3WOJ7VCQ33UTQTPTOKHS5DWRLENC3XQ, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.907647867 (567h20m5.260432975s) [noise=true , address=45.91.116.11:4117    , piece_id=C2MIQBV4JFBRAD3DOTOB775NZEMD7YVAHJ5FHRDNDBZUT7FNT6JA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.908004927 (567h20m5.260790036s) [noise=true , address=77.90.31.12:4592     , piece_id=5M4OPGMTWMWPK76RI62BYFIVOD4UFSLHVC5IA4JDGIPHUDJYT43Q, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.908367628 (567h20m5.261152726s) [noise=true , address=45.91.116.14:4191    , piece_id=O74UZOE7U5KFCFU5XQHGVMMWN65CV6WOFHWOLTIDJLFJHU6XSVFA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.908711954 (567h20m5.261497062s) [noise=true , address=45.91.116.6:4975     , piece_id=XZPUJCBJHQKMFMRC5K2EC2VFC2GBKUWP4NJ5BHXTC62WEUBQXN7Q, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.909094382 (567h20m5.26187948s) [noise=true , address=45.91.116.12:4227    , piece_id=H7UHE6ZHCOFBUFDSH4GMB3LQURR5EXM3J2LPN2763HQ7DHT2Z72Q, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.909396058 (567h20m5.262181156s) [noise=true , address=45.91.116.8:4045     , piece_id=GKLKTCUBM22LS6W2ZMKI3GRJHSGW4XJVJZMMQH7PCSM7BXGWTUYQ, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.909938486 (567h20m5.262723584s) [noise=true , address=77.90.31.19:4843     , piece_id=AHPHZA7SCLQYCESPMFATJYWQCZSU6X5YBDL65TNPNPD63OVUEOZQ, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.910238909 (567h20m5.263024018s) [noise=true , address=77.90.31.17:4770     , piece_id=I4JB5UT3GPD7YLVVV74OJEY62IZUSJ3GRC7ERV47BNBDOOGXWFCA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.910541658 (567h20m5.263326755s) [noise=true , address=45.91.116.14:4189    , piece_id=MX2GWMKTZNUVCI4VH2V7GQJKHKX6WQFGILSH2CJA6WB6OILIEERA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.910834568 (567h20m5.263619665s) [noise=true , address=77.90.31.10:4520     , piece_id=SLTEJQGU4MQFIGWCS4EPL3BRSILVS5JF57FJ2I46WU5OAPTN5FHA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:29.911189984 (567h20m5.263975092s) [noise=true , address=45.91.116.10:4083    , piece_id=WKRO6QK7RN6PCDL4KXFNXQVCW7EYSPJGIDS5AGHC363T26ACXU6Q, seg_pos={0 0}, upload=499296]: Done uploading piece. err: <nil>
Sep 22 05:28:32.076989427 (567h20m7.429774525s) [noise=true , address=145.79.227.14:4837   , piece_id=P2STYEHPN3EYE6HT3KTO3BYTJDLF52CIP6B42PRPIX2OBADXIJ2A, seg_pos={0 0}, upload=499296]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Sep 22 05:28:59.787412010 (567h20m35.140197139s) [noise=true , address=145.79.227.6:4258    , piece_id=KZKJ3EJGNRCDYYKILOB3R4NTQKAK44M3Y4Z6Z67B4FG6XT62P6CQ, seg_pos={0 0}, upload=499296]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:28:59.787437068 (567h20m35.140222206s) [noise=true , address=145.79.227.13:4802   , piece_id=5YWIAJRNNJRRTFUNL5PNEGWGFJBFEDK3L7SK425BBTSO7BF6LXRA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: context deadline exceeded; context deadline exceeded
Sep 22 05:28:59.787483675 (567h20m35.140268813s) [noise=true , address=145.79.227.19:4078   , piece_id=NS2NAUDD4RQV4TCUAPSPBLIMITSIAX5KPWZQ5HXZVAKYRBAZWUVA, seg_pos={0 0}, upload=499296]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:28:59.787512890 (567h20m35.140298028s) [seg_pos={0 0}, upload=499296]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 05:28:59.801215935 (567h20m35.154001063s) [upload=499296]: Done uploading segments...
Sep 22 05:28:59.801230542 (567h20m35.154015671s) [upload=499296]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.016029546s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................]
duration:
5.463751088s
FAIL (2025-09-22 05:28:53 UTC)
orbiter warning: operation: Commit
uplink: metaclient: context deadline exceeded
Duration: 5.46377371s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
9 out of 60 is failed (threshold=3) [..X........X..................X...X.....X....XXX..........X.]
duration:
422.316601ms
FAIL (2025-09-22 05:28:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 422.332501ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip:
4 out of 60 is failed (threshold=3) [.............................X.X..X..................X......]
duration:
403.348487ms
FAIL (2025-09-22 05:28:41 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment...
Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment.
Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment
Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments...
Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op
Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op
Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments...
Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 403.372571ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
9 out of 60 is failed (threshold=3) [...X........X..................X...X.....X....XXX..........X]
duration:
30.000479864s
FAIL (2025-09-22 05:28:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000497817s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
11 out of 60 is failed (threshold=3) [..............X...X.......X.X.......XXX.X.....X.......X.X...]
duration:
395.622006ms
FAIL (2025-09-22 05:28:00 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 05:25:07.725497873 (567h16m43.078283011s) [upload=499250]: Starting upload
Sep 22 05:25:07.725595486 (567h16m43.078380614s) [upload=499250]: Done starting upload
Sep 22 05:25:07.725677791 (567h16m43.078462919s) [upload=499250]: Uploading segments...
Sep 22 05:25:07.726021706 (567h16m43.078806845s) [upload=499250]: Got next segment. Inline: false
Sep 22 05:25:07.726077491 (567h16m43.07886262s) [upload=499250]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:25:07.726083202 (567h16m43.07886833s) [upload=499250]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:25:07.748612770 (567h16m43.101397899s) [seg_pos={0 0}, upload=499250]: Begin upload segment...
Sep 22 05:25:07.748650050 (567h16m43.101435179s) [seg_pos={0 0}, upload=499250]: Done begin upload segment.
Sep 22 05:25:07.748673835 (567h16m43.101458963s) [noise=true , address=77.90.31.16:4733     , piece_id=Z67HRT4MKSBSQSAQTAZQC6VELTHWE5HMDDT3AKPYH5YO4RCCNBUQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.748685728 (567h16m43.101470855s) [noise=true , address=77.90.31.10:4520     , piece_id=IEQ4CBKMFTWXZEXQXA6ZNSF2DHQ7WFF5MBZZM3HRDRS5RCN6SWJA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.749171399 (567h16m43.101956537s) [noise=true , address=77.90.31.21:4877     , piece_id=2RFO77NMHXMCUFFEWAK5WIGSRANBDAQWQLBV657SFHCHQPZPVXUA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.749217946 (567h16m43.102003075s) [noise=true , address=77.90.31.19:4841     , piece_id=U5BNXFA7A5GDHQDVJOJ2B7RT5VDUNCK4WJVSNOKXZ3MTY2C72S2Q, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.749558946 (567h16m43.102344084s) [noise=true , address=45.91.116.12:4226    , piece_id=NWWG2WEY333IQD2RBHXTGFPDD3T3AS23BA4KKE5OWPBPHEKLPDCQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.749682808 (567h16m43.102467936s) [noise=true , address=145.79.227.17:4972   , piece_id=YLCDONQODZMB6V3WHKT5YTRNRSD7HVAUMHQGF72YGHONTYMBRLJQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.750008239 (567h16m43.102793368s) [noise=true , address=45.91.116.5:4299     , piece_id=KK4NDRW3FBYMY23KGRHRVQSQF7CCVIJ5X6HL76PHDDSQZKXP7MRA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.750119629 (567h16m43.102904756s) [noise=true , address=45.91.116.13:4153    , piece_id=SVMKSGKJXRRV7AZSDWTGPQJJF5T7OTW7YTT422YCIDKAEB3E3WAQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.750542221 (567h16m43.103327349s) [noise=true , address=77.90.31.17:4770     , piece_id=2A7FJEJEOXMNQJGKXOW5MU7HV5B57SALK3UYFAIJLYYDYTLIPDRQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.751021030 (567h16m43.103806168s) [noise=true , address=45.91.116.4:4939     , piece_id=YB7IHEDX3OVTF2ADI5UQVVCM37DJ5AM54GKNEEHKP3YONEO3FPVQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.751386936 (567h16m43.104172065s) [noise=true , address=45.91.116.10:4082    , piece_id=DBSKEZWBTSM7FMYH5BN5IDS5SUFDEND7IZ42WVERMHIOA3FMN6DA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.751885813 (567h16m43.104670951s) [noise=true , address=145.79.227.3:4151    , piece_id=YM6JM6DSDJIMWCBZKDSWXVEW6SYTZLX5X6HVHDMLGF3OWYYB2GWA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.752311391 (567h16m43.10509652s) [noise=true , address=145.79.227.14:4838   , piece_id=YDVQ2PPO42TK6VECRTS672G33VJWT2FX6KPSRF56TA2UNTJMQA2Q, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.752667840 (567h16m43.105452968s) [noise=true , address=145.79.227.5:4221    , piece_id=NZFW5XGOLIKGF22RPTV6J4RFS22ZZDY5QQ6QZ66MR5P4IAQACODQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.753048575 (567h16m43.105833703s) [noise=true , address=77.90.31.12:4590     , piece_id=2ZI5MJOGB6QGACI4ZIDD2Y7MSLCSXYPBQN3OBXGSBZGTKPC7D76Q, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.753413008 (567h16m43.106198136s) [noise=true , address=45.91.116.14:4190    , piece_id=2VZ4LLY2QMW4DMNXJYXXMOCH6BGD2DCMT52SMG6J3QBCJZREGMWA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.867017086 (567h16m43.219802214s) [upload=499250]: Next returned nil segment
Sep 22 05:25:07.867037003 (567h16m43.219822172s) [upload=499250]: Waiting for error group managing segments...
Sep 22 05:25:07.875172330 (567h16m43.227957468s) [noise=true , address=77.90.31.16:4733     , piece_id=Z67HRT4MKSBSQSAQTAZQC6VELTHWE5HMDDT3AKPYH5YO4RCCNBUQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.875621913 (567h16m43.228407041s) [noise=true , address=77.90.31.10:4520     , piece_id=IEQ4CBKMFTWXZEXQXA6ZNSF2DHQ7WFF5MBZZM3HRDRS5RCN6SWJA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.875864970 (567h16m43.228650098s) [noise=true , address=77.90.31.21:4877     , piece_id=2RFO77NMHXMCUFFEWAK5WIGSRANBDAQWQLBV657SFHCHQPZPVXUA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.876396136 (567h16m43.229181264s) [noise=true , address=45.91.116.5:4299     , piece_id=KK4NDRW3FBYMY23KGRHRVQSQF7CCVIJ5X6HL76PHDDSQZKXP7MRA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.876604978 (567h16m43.229390106s) [noise=true , address=45.91.116.14:4190    , piece_id=2VZ4LLY2QMW4DMNXJYXXMOCH6BGD2DCMT52SMG6J3QBCJZREGMWA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.877065974 (567h16m43.229851101s) [noise=true , address=45.91.116.12:4226    , piece_id=NWWG2WEY333IQD2RBHXTGFPDD3T3AS23BA4KKE5OWPBPHEKLPDCQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.877272922 (567h16m43.230058049s) [noise=true , address=45.91.116.10:4082    , piece_id=DBSKEZWBTSM7FMYH5BN5IDS5SUFDEND7IZ42WVERMHIOA3FMN6DA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.877804229 (567h16m43.230589357s) [noise=true , address=77.90.31.17:4770     , piece_id=2A7FJEJEOXMNQJGKXOW5MU7HV5B57SALK3UYFAIJLYYDYTLIPDRQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.877973847 (567h16m43.230758976s) [noise=true , address=45.91.116.13:4153    , piece_id=SVMKSGKJXRRV7AZSDWTGPQJJF5T7OTW7YTT422YCIDKAEB3E3WAQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.878370711 (567h16m43.23115584s) [noise=true , address=45.91.116.4:4939     , piece_id=YB7IHEDX3OVTF2ADI5UQVVCM37DJ5AM54GKNEEHKP3YONEO3FPVQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.878525522 (567h16m43.23131065s) [noise=true , address=77.90.31.12:4590     , piece_id=2ZI5MJOGB6QGACI4ZIDD2Y7MSLCSXYPBQN3OBXGSBZGTKPC7D76Q, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.879039868 (567h16m43.231825005s) [noise=true , address=77.90.31.19:4841     , piece_id=U5BNXFA7A5GDHQDVJOJ2B7RT5VDUNCK4WJVSNOKXZ3MTY2C72S2Q, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:09.934935235 (567h16m45.287720363s) [noise=true , address=145.79.227.3:4151    , piece_id=YM6JM6DSDJIMWCBZKDSWXVEW6SYTZLX5X6HVHDMLGF3OWYYB2GWA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op
Sep 22 05:25:11.406580507 (567h16m46.759365615s) [noise=true , address=145.79.227.14:4838   , piece_id=YDVQ2PPO42TK6VECRTS672G33VJWT2FX6KPSRF56TA2UNTJMQA2Q, seg_pos={0 0}, upload=499250]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: connect: no route to host; manager closed: conn closed: op
Sep 22 05:25:37.711986122 (567h17m13.06477123s) [noise=true , address=145.79.227.17:4972   , piece_id=YLCDONQODZMB6V3WHKT5YTRNRSD7HVAUMHQGF72YGHONTYMBRLJQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:37.711998796 (567h17m13.064783904s) [noise=true , address=145.79.227.5:4221    , piece_id=NZFW5XGOLIKGF22RPTV6J4RFS22ZZDY5QQ6QZ66MR5P4IAQACODQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:37.712012231 (567h17m13.064797329s) [seg_pos={0 0}, upload=499250]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 05:25:37.729996459 (567h17m13.082781558s) [upload=499250]: Done uploading segments...
Sep 22 05:25:37.730015695 (567h17m13.082800803s) [upload=499250]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 395.656642ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................]
duration:
5.739500454s
FAIL (2025-09-22 05:27:46 UTC)
orbiter warning: operation: Commit
uplink: metaclient: context deadline exceeded
Duration: 5.739523918s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................]
duration:
6.134187747s
FAIL (2025-09-22 05:27:31 UTC)
orbiter warning: operation: Commit
uplink: metaclient: context deadline exceeded
Duration: 6.134217273s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip:
4 out of 60 is failed (threshold=3) [..............................X.X..X..................X.....]
duration:
649.306446ms
FAIL (2025-09-22 05:27:29 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment...
Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment.
Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment
Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments...
Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op
Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op
Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments...
Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 649.330873ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
8 out of 60 is failed (threshold=3) [....X........X..................X...X.....X....XXX..........]
duration:
495.751668ms
FAIL (2025-09-22 05:27:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 495.767579ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
8 out of 60 is failed (threshold=3) [.....X........X..................X...X.....X....XXX.........]
duration:
443.899785ms
FAIL (2025-09-22 05:27:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 443.917498ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
11 out of 60 is failed (threshold=3) [...............X...X.......X.X.......XXX.X.....X.......X.X..]
duration:
554.434101ms
FAIL (2025-09-22 05:26:47 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 05:25:07.725497873 (567h16m43.078283011s) [upload=499250]: Starting upload
Sep 22 05:25:07.725595486 (567h16m43.078380614s) [upload=499250]: Done starting upload
Sep 22 05:25:07.725677791 (567h16m43.078462919s) [upload=499250]: Uploading segments...
Sep 22 05:25:07.726021706 (567h16m43.078806845s) [upload=499250]: Got next segment. Inline: false
Sep 22 05:25:07.726077491 (567h16m43.07886262s) [upload=499250]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:25:07.726083202 (567h16m43.07886833s) [upload=499250]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:25:07.748612770 (567h16m43.101397899s) [seg_pos={0 0}, upload=499250]: Begin upload segment...
Sep 22 05:25:07.748650050 (567h16m43.101435179s) [seg_pos={0 0}, upload=499250]: Done begin upload segment.
Sep 22 05:25:07.748673835 (567h16m43.101458963s) [noise=true , address=77.90.31.16:4733     , piece_id=Z67HRT4MKSBSQSAQTAZQC6VELTHWE5HMDDT3AKPYH5YO4RCCNBUQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.748685728 (567h16m43.101470855s) [noise=true , address=77.90.31.10:4520     , piece_id=IEQ4CBKMFTWXZEXQXA6ZNSF2DHQ7WFF5MBZZM3HRDRS5RCN6SWJA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.749171399 (567h16m43.101956537s) [noise=true , address=77.90.31.21:4877     , piece_id=2RFO77NMHXMCUFFEWAK5WIGSRANBDAQWQLBV657SFHCHQPZPVXUA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.749217946 (567h16m43.102003075s) [noise=true , address=77.90.31.19:4841     , piece_id=U5BNXFA7A5GDHQDVJOJ2B7RT5VDUNCK4WJVSNOKXZ3MTY2C72S2Q, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.749558946 (567h16m43.102344084s) [noise=true , address=45.91.116.12:4226    , piece_id=NWWG2WEY333IQD2RBHXTGFPDD3T3AS23BA4KKE5OWPBPHEKLPDCQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.749682808 (567h16m43.102467936s) [noise=true , address=145.79.227.17:4972   , piece_id=YLCDONQODZMB6V3WHKT5YTRNRSD7HVAUMHQGF72YGHONTYMBRLJQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.750008239 (567h16m43.102793368s) [noise=true , address=45.91.116.5:4299     , piece_id=KK4NDRW3FBYMY23KGRHRVQSQF7CCVIJ5X6HL76PHDDSQZKXP7MRA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.750119629 (567h16m43.102904756s) [noise=true , address=45.91.116.13:4153    , piece_id=SVMKSGKJXRRV7AZSDWTGPQJJF5T7OTW7YTT422YCIDKAEB3E3WAQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.750542221 (567h16m43.103327349s) [noise=true , address=77.90.31.17:4770     , piece_id=2A7FJEJEOXMNQJGKXOW5MU7HV5B57SALK3UYFAIJLYYDYTLIPDRQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.751021030 (567h16m43.103806168s) [noise=true , address=45.91.116.4:4939     , piece_id=YB7IHEDX3OVTF2ADI5UQVVCM37DJ5AM54GKNEEHKP3YONEO3FPVQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.751386936 (567h16m43.104172065s) [noise=true , address=45.91.116.10:4082    , piece_id=DBSKEZWBTSM7FMYH5BN5IDS5SUFDEND7IZ42WVERMHIOA3FMN6DA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.751885813 (567h16m43.104670951s) [noise=true , address=145.79.227.3:4151    , piece_id=YM6JM6DSDJIMWCBZKDSWXVEW6SYTZLX5X6HVHDMLGF3OWYYB2GWA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.752311391 (567h16m43.10509652s) [noise=true , address=145.79.227.14:4838   , piece_id=YDVQ2PPO42TK6VECRTS672G33VJWT2FX6KPSRF56TA2UNTJMQA2Q, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.752667840 (567h16m43.105452968s) [noise=true , address=145.79.227.5:4221    , piece_id=NZFW5XGOLIKGF22RPTV6J4RFS22ZZDY5QQ6QZ66MR5P4IAQACODQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.753048575 (567h16m43.105833703s) [noise=true , address=77.90.31.12:4590     , piece_id=2ZI5MJOGB6QGACI4ZIDD2Y7MSLCSXYPBQN3OBXGSBZGTKPC7D76Q, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.753413008 (567h16m43.106198136s) [noise=true , address=45.91.116.14:4190    , piece_id=2VZ4LLY2QMW4DMNXJYXXMOCH6BGD2DCMT52SMG6J3QBCJZREGMWA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.867017086 (567h16m43.219802214s) [upload=499250]: Next returned nil segment
Sep 22 05:25:07.867037003 (567h16m43.219822172s) [upload=499250]: Waiting for error group managing segments...
Sep 22 05:25:07.875172330 (567h16m43.227957468s) [noise=true , address=77.90.31.16:4733     , piece_id=Z67HRT4MKSBSQSAQTAZQC6VELTHWE5HMDDT3AKPYH5YO4RCCNBUQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.875621913 (567h16m43.228407041s) [noise=true , address=77.90.31.10:4520     , piece_id=IEQ4CBKMFTWXZEXQXA6ZNSF2DHQ7WFF5MBZZM3HRDRS5RCN6SWJA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.875864970 (567h16m43.228650098s) [noise=true , address=77.90.31.21:4877     , piece_id=2RFO77NMHXMCUFFEWAK5WIGSRANBDAQWQLBV657SFHCHQPZPVXUA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.876396136 (567h16m43.229181264s) [noise=true , address=45.91.116.5:4299     , piece_id=KK4NDRW3FBYMY23KGRHRVQSQF7CCVIJ5X6HL76PHDDSQZKXP7MRA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.876604978 (567h16m43.229390106s) [noise=true , address=45.91.116.14:4190    , piece_id=2VZ4LLY2QMW4DMNXJYXXMOCH6BGD2DCMT52SMG6J3QBCJZREGMWA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.877065974 (567h16m43.229851101s) [noise=true , address=45.91.116.12:4226    , piece_id=NWWG2WEY333IQD2RBHXTGFPDD3T3AS23BA4KKE5OWPBPHEKLPDCQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.877272922 (567h16m43.230058049s) [noise=true , address=45.91.116.10:4082    , piece_id=DBSKEZWBTSM7FMYH5BN5IDS5SUFDEND7IZ42WVERMHIOA3FMN6DA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.877804229 (567h16m43.230589357s) [noise=true , address=77.90.31.17:4770     , piece_id=2A7FJEJEOXMNQJGKXOW5MU7HV5B57SALK3UYFAIJLYYDYTLIPDRQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.877973847 (567h16m43.230758976s) [noise=true , address=45.91.116.13:4153    , piece_id=SVMKSGKJXRRV7AZSDWTGPQJJF5T7OTW7YTT422YCIDKAEB3E3WAQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.878370711 (567h16m43.23115584s) [noise=true , address=45.91.116.4:4939     , piece_id=YB7IHEDX3OVTF2ADI5UQVVCM37DJ5AM54GKNEEHKP3YONEO3FPVQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.878525522 (567h16m43.23131065s) [noise=true , address=77.90.31.12:4590     , piece_id=2ZI5MJOGB6QGACI4ZIDD2Y7MSLCSXYPBQN3OBXGSBZGTKPC7D76Q, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.879039868 (567h16m43.231825005s) [noise=true , address=77.90.31.19:4841     , piece_id=U5BNXFA7A5GDHQDVJOJ2B7RT5VDUNCK4WJVSNOKXZ3MTY2C72S2Q, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:09.934935235 (567h16m45.287720363s) [noise=true , address=145.79.227.3:4151    , piece_id=YM6JM6DSDJIMWCBZKDSWXVEW6SYTZLX5X6HVHDMLGF3OWYYB2GWA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op
Sep 22 05:25:11.406580507 (567h16m46.759365615s) [noise=true , address=145.79.227.14:4838   , piece_id=YDVQ2PPO42TK6VECRTS672G33VJWT2FX6KPSRF56TA2UNTJMQA2Q, seg_pos={0 0}, upload=499250]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: connect: no route to host; manager closed: conn closed: op
Sep 22 05:25:37.711986122 (567h17m13.06477123s) [noise=true , address=145.79.227.17:4972   , piece_id=YLCDONQODZMB6V3WHKT5YTRNRSD7HVAUMHQGF72YGHONTYMBRLJQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:37.711998796 (567h17m13.064783904s) [noise=true , address=145.79.227.5:4221    , piece_id=NZFW5XGOLIKGF22RPTV6J4RFS22ZZDY5QQ6QZ66MR5P4IAQACODQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:37.712012231 (567h17m13.064797329s) [seg_pos={0 0}, upload=499250]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 05:25:37.729996459 (567h17m13.082781558s) [upload=499250]: Done uploading segments...
Sep 22 05:25:37.730015695 (567h17m13.082800803s) [upload=499250]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 554.458716ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................]
duration:
5.644643627s
FAIL (2025-09-22 05:26:46 UTC)
orbiter warning: operation: Commit
uplink: metaclient: context deadline exceeded
Duration: 5.644669125s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
8 out of 60 is failed (threshold=3) [......X........X..................X...X.....X....XXX........]
duration:
400.708042ms
FAIL (2025-09-22 05:26:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 400.721276ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip:
4 out of 60 is failed (threshold=3) [...............................X.X..X..................X....]
duration:
415.307187ms
FAIL (2025-09-22 05:26:40 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment...
Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment.
Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment
Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments...
Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op
Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op
Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments...
Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 415.333767ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip:
4 out of 60 is failed (threshold=3) [................................X.X..X..................X...]
duration:
408.347066ms
FAIL (2025-09-22 05:26:23 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment...
Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment.
Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment
Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments...
Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op
Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op
Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments...
Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 408.368947ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
11 out of 60 is failed (threshold=3) [................X...X.......X.X.......XXX.X.....X.......X.X.]
duration:
416.651761ms
FAIL (2025-09-22 05:26:07 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 05:25:07.725497873 (567h16m43.078283011s) [upload=499250]: Starting upload
Sep 22 05:25:07.725595486 (567h16m43.078380614s) [upload=499250]: Done starting upload
Sep 22 05:25:07.725677791 (567h16m43.078462919s) [upload=499250]: Uploading segments...
Sep 22 05:25:07.726021706 (567h16m43.078806845s) [upload=499250]: Got next segment. Inline: false
Sep 22 05:25:07.726077491 (567h16m43.07886262s) [upload=499250]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:25:07.726083202 (567h16m43.07886833s) [upload=499250]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:25:07.748612770 (567h16m43.101397899s) [seg_pos={0 0}, upload=499250]: Begin upload segment...
Sep 22 05:25:07.748650050 (567h16m43.101435179s) [seg_pos={0 0}, upload=499250]: Done begin upload segment.
Sep 22 05:25:07.748673835 (567h16m43.101458963s) [noise=true , address=77.90.31.16:4733     , piece_id=Z67HRT4MKSBSQSAQTAZQC6VELTHWE5HMDDT3AKPYH5YO4RCCNBUQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.748685728 (567h16m43.101470855s) [noise=true , address=77.90.31.10:4520     , piece_id=IEQ4CBKMFTWXZEXQXA6ZNSF2DHQ7WFF5MBZZM3HRDRS5RCN6SWJA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.749171399 (567h16m43.101956537s) [noise=true , address=77.90.31.21:4877     , piece_id=2RFO77NMHXMCUFFEWAK5WIGSRANBDAQWQLBV657SFHCHQPZPVXUA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.749217946 (567h16m43.102003075s) [noise=true , address=77.90.31.19:4841     , piece_id=U5BNXFA7A5GDHQDVJOJ2B7RT5VDUNCK4WJVSNOKXZ3MTY2C72S2Q, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.749558946 (567h16m43.102344084s) [noise=true , address=45.91.116.12:4226    , piece_id=NWWG2WEY333IQD2RBHXTGFPDD3T3AS23BA4KKE5OWPBPHEKLPDCQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.749682808 (567h16m43.102467936s) [noise=true , address=145.79.227.17:4972   , piece_id=YLCDONQODZMB6V3WHKT5YTRNRSD7HVAUMHQGF72YGHONTYMBRLJQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.750008239 (567h16m43.102793368s) [noise=true , address=45.91.116.5:4299     , piece_id=KK4NDRW3FBYMY23KGRHRVQSQF7CCVIJ5X6HL76PHDDSQZKXP7MRA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.750119629 (567h16m43.102904756s) [noise=true , address=45.91.116.13:4153    , piece_id=SVMKSGKJXRRV7AZSDWTGPQJJF5T7OTW7YTT422YCIDKAEB3E3WAQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.750542221 (567h16m43.103327349s) [noise=true , address=77.90.31.17:4770     , piece_id=2A7FJEJEOXMNQJGKXOW5MU7HV5B57SALK3UYFAIJLYYDYTLIPDRQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.751021030 (567h16m43.103806168s) [noise=true , address=45.91.116.4:4939     , piece_id=YB7IHEDX3OVTF2ADI5UQVVCM37DJ5AM54GKNEEHKP3YONEO3FPVQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.751386936 (567h16m43.104172065s) [noise=true , address=45.91.116.10:4082    , piece_id=DBSKEZWBTSM7FMYH5BN5IDS5SUFDEND7IZ42WVERMHIOA3FMN6DA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.751885813 (567h16m43.104670951s) [noise=true , address=145.79.227.3:4151    , piece_id=YM6JM6DSDJIMWCBZKDSWXVEW6SYTZLX5X6HVHDMLGF3OWYYB2GWA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.752311391 (567h16m43.10509652s) [noise=true , address=145.79.227.14:4838   , piece_id=YDVQ2PPO42TK6VECRTS672G33VJWT2FX6KPSRF56TA2UNTJMQA2Q, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.752667840 (567h16m43.105452968s) [noise=true , address=145.79.227.5:4221    , piece_id=NZFW5XGOLIKGF22RPTV6J4RFS22ZZDY5QQ6QZ66MR5P4IAQACODQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.753048575 (567h16m43.105833703s) [noise=true , address=77.90.31.12:4590     , piece_id=2ZI5MJOGB6QGACI4ZIDD2Y7MSLCSXYPBQN3OBXGSBZGTKPC7D76Q, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.753413008 (567h16m43.106198136s) [noise=true , address=45.91.116.14:4190    , piece_id=2VZ4LLY2QMW4DMNXJYXXMOCH6BGD2DCMT52SMG6J3QBCJZREGMWA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.867017086 (567h16m43.219802214s) [upload=499250]: Next returned nil segment
Sep 22 05:25:07.867037003 (567h16m43.219822172s) [upload=499250]: Waiting for error group managing segments...
Sep 22 05:25:07.875172330 (567h16m43.227957468s) [noise=true , address=77.90.31.16:4733     , piece_id=Z67HRT4MKSBSQSAQTAZQC6VELTHWE5HMDDT3AKPYH5YO4RCCNBUQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.875621913 (567h16m43.228407041s) [noise=true , address=77.90.31.10:4520     , piece_id=IEQ4CBKMFTWXZEXQXA6ZNSF2DHQ7WFF5MBZZM3HRDRS5RCN6SWJA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.875864970 (567h16m43.228650098s) [noise=true , address=77.90.31.21:4877     , piece_id=2RFO77NMHXMCUFFEWAK5WIGSRANBDAQWQLBV657SFHCHQPZPVXUA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.876396136 (567h16m43.229181264s) [noise=true , address=45.91.116.5:4299     , piece_id=KK4NDRW3FBYMY23KGRHRVQSQF7CCVIJ5X6HL76PHDDSQZKXP7MRA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.876604978 (567h16m43.229390106s) [noise=true , address=45.91.116.14:4190    , piece_id=2VZ4LLY2QMW4DMNXJYXXMOCH6BGD2DCMT52SMG6J3QBCJZREGMWA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.877065974 (567h16m43.229851101s) [noise=true , address=45.91.116.12:4226    , piece_id=NWWG2WEY333IQD2RBHXTGFPDD3T3AS23BA4KKE5OWPBPHEKLPDCQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.877272922 (567h16m43.230058049s) [noise=true , address=45.91.116.10:4082    , piece_id=DBSKEZWBTSM7FMYH5BN5IDS5SUFDEND7IZ42WVERMHIOA3FMN6DA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.877804229 (567h16m43.230589357s) [noise=true , address=77.90.31.17:4770     , piece_id=2A7FJEJEOXMNQJGKXOW5MU7HV5B57SALK3UYFAIJLYYDYTLIPDRQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.877973847 (567h16m43.230758976s) [noise=true , address=45.91.116.13:4153    , piece_id=SVMKSGKJXRRV7AZSDWTGPQJJF5T7OTW7YTT422YCIDKAEB3E3WAQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.878370711 (567h16m43.23115584s) [noise=true , address=45.91.116.4:4939     , piece_id=YB7IHEDX3OVTF2ADI5UQVVCM37DJ5AM54GKNEEHKP3YONEO3FPVQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.878525522 (567h16m43.23131065s) [noise=true , address=77.90.31.12:4590     , piece_id=2ZI5MJOGB6QGACI4ZIDD2Y7MSLCSXYPBQN3OBXGSBZGTKPC7D76Q, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.879039868 (567h16m43.231825005s) [noise=true , address=77.90.31.19:4841     , piece_id=U5BNXFA7A5GDHQDVJOJ2B7RT5VDUNCK4WJVSNOKXZ3MTY2C72S2Q, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:09.934935235 (567h16m45.287720363s) [noise=true , address=145.79.227.3:4151    , piece_id=YM6JM6DSDJIMWCBZKDSWXVEW6SYTZLX5X6HVHDMLGF3OWYYB2GWA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op
Sep 22 05:25:11.406580507 (567h16m46.759365615s) [noise=true , address=145.79.227.14:4838   , piece_id=YDVQ2PPO42TK6VECRTS672G33VJWT2FX6KPSRF56TA2UNTJMQA2Q, seg_pos={0 0}, upload=499250]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: connect: no route to host; manager closed: conn closed: op
Sep 22 05:25:37.711986122 (567h17m13.06477123s) [noise=true , address=145.79.227.17:4972   , piece_id=YLCDONQODZMB6V3WHKT5YTRNRSD7HVAUMHQGF72YGHONTYMBRLJQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:37.711998796 (567h17m13.064783904s) [noise=true , address=145.79.227.5:4221    , piece_id=NZFW5XGOLIKGF22RPTV6J4RFS22ZZDY5QQ6QZ66MR5P4IAQACODQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:37.712012231 (567h17m13.064797329s) [seg_pos={0 0}, upload=499250]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 05:25:37.729996459 (567h17m13.082781558s) [upload=499250]: Done uploading segments...
Sep 22 05:25:37.730015695 (567h17m13.082800803s) [upload=499250]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 416.671318ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................]
duration:
5.587049068s
FAIL (2025-09-22 05:26:06 UTC)
orbiter warning: operation: Commit
uplink: metaclient: context deadline exceeded
Duration: 5.587073574s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip:
4 out of 60 is failed (threshold=3) [.................................X.X..X..................X..]
duration:
425.161361ms
FAIL (2025-09-22 05:26:01 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment...
Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment.
Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment
Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments...
Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op
Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op
Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments...
Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 425.205915ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
8 out of 60 is failed (threshold=3) [.......X........X..................X...X.....X....XXX.......]
duration:
678.355409ms
FAIL (2025-09-22 05:25:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 678.374906ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
11 out of 60 is failed (threshold=3) [.................X...X.......X.X.......XXX.X.....X.......X.X]
duration:
30.019905088s
FAIL (2025-09-22 05:25:37 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 05:25:07.725497873 (567h16m43.078283011s) [upload=499250]: Starting upload
Sep 22 05:25:07.725595486 (567h16m43.078380614s) [upload=499250]: Done starting upload
Sep 22 05:25:07.725677791 (567h16m43.078462919s) [upload=499250]: Uploading segments...
Sep 22 05:25:07.726021706 (567h16m43.078806845s) [upload=499250]: Got next segment. Inline: false
Sep 22 05:25:07.726077491 (567h16m43.07886262s) [upload=499250]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:25:07.726083202 (567h16m43.07886833s) [upload=499250]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:25:07.748612770 (567h16m43.101397899s) [seg_pos={0 0}, upload=499250]: Begin upload segment...
Sep 22 05:25:07.748650050 (567h16m43.101435179s) [seg_pos={0 0}, upload=499250]: Done begin upload segment.
Sep 22 05:25:07.748673835 (567h16m43.101458963s) [noise=true , address=77.90.31.16:4733     , piece_id=Z67HRT4MKSBSQSAQTAZQC6VELTHWE5HMDDT3AKPYH5YO4RCCNBUQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.748685728 (567h16m43.101470855s) [noise=true , address=77.90.31.10:4520     , piece_id=IEQ4CBKMFTWXZEXQXA6ZNSF2DHQ7WFF5MBZZM3HRDRS5RCN6SWJA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.749171399 (567h16m43.101956537s) [noise=true , address=77.90.31.21:4877     , piece_id=2RFO77NMHXMCUFFEWAK5WIGSRANBDAQWQLBV657SFHCHQPZPVXUA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.749217946 (567h16m43.102003075s) [noise=true , address=77.90.31.19:4841     , piece_id=U5BNXFA7A5GDHQDVJOJ2B7RT5VDUNCK4WJVSNOKXZ3MTY2C72S2Q, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.749558946 (567h16m43.102344084s) [noise=true , address=45.91.116.12:4226    , piece_id=NWWG2WEY333IQD2RBHXTGFPDD3T3AS23BA4KKE5OWPBPHEKLPDCQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.749682808 (567h16m43.102467936s) [noise=true , address=145.79.227.17:4972   , piece_id=YLCDONQODZMB6V3WHKT5YTRNRSD7HVAUMHQGF72YGHONTYMBRLJQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.750008239 (567h16m43.102793368s) [noise=true , address=45.91.116.5:4299     , piece_id=KK4NDRW3FBYMY23KGRHRVQSQF7CCVIJ5X6HL76PHDDSQZKXP7MRA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.750119629 (567h16m43.102904756s) [noise=true , address=45.91.116.13:4153    , piece_id=SVMKSGKJXRRV7AZSDWTGPQJJF5T7OTW7YTT422YCIDKAEB3E3WAQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.750542221 (567h16m43.103327349s) [noise=true , address=77.90.31.17:4770     , piece_id=2A7FJEJEOXMNQJGKXOW5MU7HV5B57SALK3UYFAIJLYYDYTLIPDRQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.751021030 (567h16m43.103806168s) [noise=true , address=45.91.116.4:4939     , piece_id=YB7IHEDX3OVTF2ADI5UQVVCM37DJ5AM54GKNEEHKP3YONEO3FPVQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.751386936 (567h16m43.104172065s) [noise=true , address=45.91.116.10:4082    , piece_id=DBSKEZWBTSM7FMYH5BN5IDS5SUFDEND7IZ42WVERMHIOA3FMN6DA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.751885813 (567h16m43.104670951s) [noise=true , address=145.79.227.3:4151    , piece_id=YM6JM6DSDJIMWCBZKDSWXVEW6SYTZLX5X6HVHDMLGF3OWYYB2GWA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.752311391 (567h16m43.10509652s) [noise=true , address=145.79.227.14:4838   , piece_id=YDVQ2PPO42TK6VECRTS672G33VJWT2FX6KPSRF56TA2UNTJMQA2Q, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.752667840 (567h16m43.105452968s) [noise=true , address=145.79.227.5:4221    , piece_id=NZFW5XGOLIKGF22RPTV6J4RFS22ZZDY5QQ6QZ66MR5P4IAQACODQ, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.753048575 (567h16m43.105833703s) [noise=true , address=77.90.31.12:4590     , piece_id=2ZI5MJOGB6QGACI4ZIDD2Y7MSLCSXYPBQN3OBXGSBZGTKPC7D76Q, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.753413008 (567h16m43.106198136s) [noise=true , address=45.91.116.14:4190    , piece_id=2VZ4LLY2QMW4DMNXJYXXMOCH6BGD2DCMT52SMG6J3QBCJZREGMWA, seg_pos={0 0}, upload=499250]: Uploading piece...
Sep 22 05:25:07.867017086 (567h16m43.219802214s) [upload=499250]: Next returned nil segment
Sep 22 05:25:07.867037003 (567h16m43.219822172s) [upload=499250]: Waiting for error group managing segments...
Sep 22 05:25:07.875172330 (567h16m43.227957468s) [noise=true , address=77.90.31.16:4733     , piece_id=Z67HRT4MKSBSQSAQTAZQC6VELTHWE5HMDDT3AKPYH5YO4RCCNBUQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.875621913 (567h16m43.228407041s) [noise=true , address=77.90.31.10:4520     , piece_id=IEQ4CBKMFTWXZEXQXA6ZNSF2DHQ7WFF5MBZZM3HRDRS5RCN6SWJA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.875864970 (567h16m43.228650098s) [noise=true , address=77.90.31.21:4877     , piece_id=2RFO77NMHXMCUFFEWAK5WIGSRANBDAQWQLBV657SFHCHQPZPVXUA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.876396136 (567h16m43.229181264s) [noise=true , address=45.91.116.5:4299     , piece_id=KK4NDRW3FBYMY23KGRHRVQSQF7CCVIJ5X6HL76PHDDSQZKXP7MRA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.876604978 (567h16m43.229390106s) [noise=true , address=45.91.116.14:4190    , piece_id=2VZ4LLY2QMW4DMNXJYXXMOCH6BGD2DCMT52SMG6J3QBCJZREGMWA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.877065974 (567h16m43.229851101s) [noise=true , address=45.91.116.12:4226    , piece_id=NWWG2WEY333IQD2RBHXTGFPDD3T3AS23BA4KKE5OWPBPHEKLPDCQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.877272922 (567h16m43.230058049s) [noise=true , address=45.91.116.10:4082    , piece_id=DBSKEZWBTSM7FMYH5BN5IDS5SUFDEND7IZ42WVERMHIOA3FMN6DA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.877804229 (567h16m43.230589357s) [noise=true , address=77.90.31.17:4770     , piece_id=2A7FJEJEOXMNQJGKXOW5MU7HV5B57SALK3UYFAIJLYYDYTLIPDRQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.877973847 (567h16m43.230758976s) [noise=true , address=45.91.116.13:4153    , piece_id=SVMKSGKJXRRV7AZSDWTGPQJJF5T7OTW7YTT422YCIDKAEB3E3WAQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.878370711 (567h16m43.23115584s) [noise=true , address=45.91.116.4:4939     , piece_id=YB7IHEDX3OVTF2ADI5UQVVCM37DJ5AM54GKNEEHKP3YONEO3FPVQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.878525522 (567h16m43.23131065s) [noise=true , address=77.90.31.12:4590     , piece_id=2ZI5MJOGB6QGACI4ZIDD2Y7MSLCSXYPBQN3OBXGSBZGTKPC7D76Q, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:07.879039868 (567h16m43.231825005s) [noise=true , address=77.90.31.19:4841     , piece_id=U5BNXFA7A5GDHQDVJOJ2B7RT5VDUNCK4WJVSNOKXZ3MTY2C72S2Q, seg_pos={0 0}, upload=499250]: Done uploading piece. err: <nil>
Sep 22 05:25:09.934935235 (567h16m45.287720363s) [noise=true , address=145.79.227.3:4151    , piece_id=YM6JM6DSDJIMWCBZKDSWXVEW6SYTZLX5X6HVHDMLGF3OWYYB2GWA, seg_pos={0 0}, upload=499250]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op
Sep 22 05:25:11.406580507 (567h16m46.759365615s) [noise=true , address=145.79.227.14:4838   , piece_id=YDVQ2PPO42TK6VECRTS672G33VJWT2FX6KPSRF56TA2UNTJMQA2Q, seg_pos={0 0}, upload=499250]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: connect: no route to host; manager closed: conn closed: op
Sep 22 05:25:37.711986122 (567h17m13.06477123s) [noise=true , address=145.79.227.17:4972   , piece_id=YLCDONQODZMB6V3WHKT5YTRNRSD7HVAUMHQGF72YGHONTYMBRLJQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:37.711998796 (567h17m13.064783904s) [noise=true , address=145.79.227.5:4221    , piece_id=NZFW5XGOLIKGF22RPTV6J4RFS22ZZDY5QQ6QZ66MR5P4IAQACODQ, seg_pos={0 0}, upload=499250]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:37.712012231 (567h17m13.064797329s) [seg_pos={0 0}, upload=499250]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 05:25:37.729996459 (567h17m13.082781558s) [upload=499250]: Done uploading segments...
Sep 22 05:25:37.730015695 (567h17m13.082800803s) [upload=499250]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.019923643s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
8 out of 60 is failed (threshold=3) [........X........X..................X...X.....X....XXX......]
duration:
434.414434ms
FAIL (2025-09-22 05:25:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 434.432969ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip:
4 out of 60 is failed (threshold=3) [..................................X.X..X..................X.]
duration:
408.602357ms
FAIL (2025-09-22 05:25:11 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment...
Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment.
Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece...
Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment
Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments...
Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896   , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093    , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586     , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561     , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029   , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624   , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586   , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895     , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168   , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136    , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538     , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559   , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil>
Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733   , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op
Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152    , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op
Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982   , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896   , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107   , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825   , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments...
Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 408.622354ms core uplink us-select-1 us1-placement-20