Orbiter History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.15436136s FAIL (2025-09-22 05:25:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.154396135s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................X.X..X..................X] duration: 1m0.016856522s FAIL (2025-09-22 05:25:01 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 9934 (567h15m36.673385032s) [upload=499235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:24:01.320603951 (567h15m36.67338905s) [upload=499235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:24:01.344325988 (567h15m36.697111096s) [seg_pos={0 0}, upload=499235]: Begin upload segment... Sep 22 05:24:01.344389868 (567h15m36.697174966s) [seg_pos={0 0}, upload=499235]: Done begin upload segment. Sep 22 05:24:01.344399326 (567h15m36.697184434s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344412581 (567h15m36.697197679s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344474567 (567h15m36.697259665s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344712894 (567h15m36.697497992s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344738953 (567h15m36.697524061s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344964736 (567h15m36.697749845s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.344993961 (567h15m36.697779059s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345215879 (567h15m36.698000976s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345430722 (567h15m36.698215819s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345644282 (567h15m36.69842938s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.345848054 (567h15m36.698633162s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346066714 (567h15m36.698851823s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346387497 (567h15m36.699172605s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346607670 (567h15m36.699392778s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.346816642 (567h15m36.69960175s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347041865 (567h15m36.699826973s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347256778 (567h15m36.700041886s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.347697735 (567h15m36.700482833s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Uploading piece... Sep 22 05:24:01.414950461 (567h15m36.767735559s) [upload=499235]: Next returned nil segment Sep 22 05:24:01.414961502 (567h15m36.76774661s) [upload=499235]: Waiting for error group managing segments... Sep 22 05:24:01.422347973 (567h15m36.775133081s) [noise=true , address=145.79.227.33:4896 , piece_id=QMOKLQUWHGHSDFWDR7IZY6BMHHJ3TWADGSMGT6TRPVMG266LOVFA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.422665028 (567h15m36.775450126s) [noise=true , address=45.91.116.10:4093 , piece_id=BIPXKZ6MS73NJRTFOF5Y2AC2QBTLBDBJUX7CZDXRPKGTU63GNDVQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423494805 (567h15m36.776279913s) [noise=true , address=77.90.31.11:4586 , piece_id=OINTNI22SDJBJVQRV7CPZTXMQLUZF3QSIT4H4GCYFSRJ3UQX2PJA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.423635299 (567h15m36.776420407s) [noise=true , address=77.90.31.11:4561 , piece_id=A4Z3F2IYHUPXJW6XPGL2W5AZ6ML3FTZWBJ2D23C2GY2BEPLU3JBQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424085374 (567h15m36.776870482s) [noise=true , address=145.79.227.39:4029 , piece_id=SN2QGNKQSZHG74IRY2D2J7ULI5Z5HZT5EAHRDIK4FZ4ZOJPPSAMA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424273466 (567h15m36.777058574s) [noise=true , address=145.79.227.46:4624 , piece_id=Q5L7C66B6ARTZZE32WUE6YCG4SDDVOOYEZSWJJH6JQCEPUGPQU7Q, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.424401617 (567h15m36.777186715s) [noise=true , address=145.79.227.24:4586 , piece_id=HBTOQADVMWPZ2NXJ6RWIJUQKRS6V74DKMYPQABLL5JZ44JP4QPLQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.425203882 (567h15m36.777988981s) [noise=true , address=77.90.31.21:4895 , piece_id=EQI7JWUW7OANDOU2WC2CEEF45DJ2DRM6TIFDLBAXB7OMSNYT7WZQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.427714755 (567h15m36.780499863s) [noise=true , address=145.79.227.43:4168 , piece_id=6Z4UAVVRKHSKOLFGTROTRVZKSBYVJNKXUOQC4XLYKODP6RZVV7GQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.437243556 (567h15m36.790028664s) [noise=true , address=45.91.116.11:4136 , piece_id=QISYHS33KDXPH4FTPM4ASCPWDQ36TUZ2L4CFDYTHMTGOUC6AWLGA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.440363601 (567h15m36.793148709s) [noise=true , address=77.90.31.10:4538 , piece_id=JAFZFM3Y7IC4LM6XIYZO5MTRFD3IFPXU6PX6TLWELGRNYWUUE42A, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:01.939873873 (567h15m37.292658971s) [noise=true , address=66.23.193.175:4559 , piece_id=QFQ6YXNDC7ZUILECHCFMYUHJO57ODHP7ZJNMXTWBJKYAU5NEKFDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: <nil> Sep 22 05:24:17.802234801 (567h15m53.155019909s) [noise=true , address=145.79.227.11:4733 , piece_id=7JQLVSKC5ZSUDVP7ZXXX2CLJZPJK5NFMGB7QDQU7W76G5I7PCTHA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: connect: no route to host; conn closed: op Sep 22 05:25:01.306924852 (567h16m36.65970996s) [noise=true , address=145.79.227.3:4152 , piece_id=WJ66BMJSGW65ZG2FBVUD4LE5APCXFHWOITLMNOH3UPAGDFGUAVDA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:145.79.227.3:4152): protocol: rpc: dial tcp 145.79.227.3:4152: i/o timeout; conn closed: op Sep 22 05:25:01.306960940 (567h16m36.659746048s) [noise=true , address=145.79.227.17:4982 , piece_id=IUXTYHGNYVXLUNOCLUWFUG4XXN7OXTRIL4GVISQOQGU2BKTNZEBA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:145.79.227.17:4982): protocol: rpc: dial tcp 145.79.227.17:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.306989373 (567h16m36.659774471s) [noise=true , address=145.79.227.15:4896 , piece_id=SLTDLXF4V2WLVH54BH2AQH7MY6CDQW5O6SLU22LBU3YOKMK6UBFQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:145.79.227.15:4896): protocol: rpc: dial tcp 145.79.227.15:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307006856 (567h16m36.659791964s) [noise=true , address=145.79.227.19:4107 , piece_id=XB4PUXGBEQA3CIVYCTDX3AXVKST6ECBW5URDOXBHJI7QZ3WUAUXA, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:145.79.227.19:4107): protocol: rpc: dial tcp 145.79.227.19:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:25:01.307076026 (567h16m36.659861124s) [noise=true , address=145.79.227.13:4825 , piece_id=TZPZQQR5RZCLH4S2C3WA7OTTLD5B452QBGTC3QYIZGQLP2YMWJYQ, seg_pos={0 0}, upload=499235]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 05:25:01.307090433 (567h16m36.659875541s) [seg_pos={0 0}, upload=499235]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:25:01.321172108 (567h16m36.673957206s) [upload=499235]: Done uploading segments... Sep 22 05:25:01.321194440 (567h16m36.673979548s) [upload=499235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.016870499s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.260013388s FAIL (2025-09-22 05:24:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.260033426s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................X...X.......X.X.......XXX.X.....X.......X.] duration: 453.333709ms FAIL (2025-09-22 05:24:30 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Sep 22 05:23:55.305745495 (567h15m30.658530594s) [upload=499231]: Starting upload Sep 22 05:23:55.305810838 (567h15m30.658595936s) [upload=499231]: Done starting upload Sep 22 05:23:55.305870911 (567h15m30.658656019s) [upload=499231]: Uploading segments... Sep 22 05:23:55.306056390 (567h15m30.658841508s) [upload=499231]: Got next segment. Inline: false Sep 22 05:23:55.306324312 (567h15m30.65910942s) [upload=499231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:23:55.306330464 (567h15m30.659115562s) [upload=499231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:23:55.329363728 (567h15m30.682148826s) [seg_pos={0 0}, upload=499231]: Begin upload segment... Sep 22 05:23:55.329392612 (567h15m30.68217771s) [seg_pos={0 0}, upload=499231]: Done begin upload segment. Sep 22 05:23:55.329412229 (567h15m30.682197327s) [noise=true , address=145.79.227.19:4078 , piece_id=27WKGKIEYCKOGF32AVABPOMR2LZAJWA6H6XDNRCJIXWYQOG7BFYA, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.329495375 (567h15m30.682280483s) [noise=true , address=45.91.116.12:4226 , piece_id=455PMCHWJV5N6GESEP366YVGF4QS7HRT37CV45NODERPDLC5N4SA, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.329724985 (567h15m30.682510093s) [noise=true , address=145.79.227.7:4295 , piece_id=HRPBBJQEOKX46PKKQYYQ3OSBKZRQ6FG3UF4LA3ADDRZ4RY4HU4IA, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.329833008 (567h15m30.682618106s) [noise=true , address=145.79.227.14:4839 , piece_id=YNLJDUMP2MZBOO466LOXZNXE47OEJPGMZ7XZGZRODJMVZGXQWCNQ, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.329991756 (567h15m30.682776864s) [noise=true , address=77.90.31.21:4879 , piece_id=ZZ2CDZCY3YGTWZVA5NKJ7GVVPIRKYRZ6L4S36DH2CIF5FWJ7PGAQ, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.330098596 (567h15m30.682883704s) [noise=true , address=145.79.227.3:4150 , piece_id=ODGIYTQ5KSG23ABMEPF25U2V2ZPBU7TUSM5J4BUQMHHCCYUSHTZQ, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.330230425 (567h15m30.683015522s) [noise=true , address=45.91.116.13:4153 , piece_id=UVAHQBORHD4BCN7TVFWJMZJ7IHK6EQAUHPTKF4L3SJKRGZ5XCUWQ, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.330332817 (567h15m30.683117914s) [noise=true , address=145.79.227.5:4223 , piece_id=6J3B7KY7UXL5Q54CYWOW5WTPMGCX3SIVDU6Y3KIWSIMIOY7ZFLGQ, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.330469012 (567h15m30.68325412s) [noise=true , address=77.90.31.21:4878 , piece_id=Q2E6VRL6B4ACWIHPY5TBOYVNC64YI6OM52N62IR36ZZDIVOPXLCA, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.330554843 (567h15m30.683339941s) [noise=true , address=145.79.227.6:4258 , piece_id=D2X5ABUMXHX2CEIPD2ZSH3XUJONB6NM54KM734EFMXZZSKI4AEEQ, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.330709423 (567h15m30.683494531s) [noise=true , address=77.90.31.15:4699 , piece_id=5SDUJTAZDJMLLZD4UW2IYKT6XH5PUPZESK52INKKWWJ7TCDO6WUA, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.330767873 (567h15m30.68355297s) [noise=true , address=77.90.31.19:4841 , piece_id=SE5OYVNORL2NZT6N4G2C45YBJSHJJEKSXMTWGOGIJZMI7IYHEYRA, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.330949173 (567h15m30.683734281s) [noise=true , address=45.91.116.10:4081 , piece_id=LNTHUHRZVQ344JQSSJU7K26K2KNWCCNXKE4KIXOI36QFJYSUJVKA, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.331003133 (567h15m30.683788241s) [noise=true , address=77.90.31.12:4590 , piece_id=XJ45LQOCHPMUJAUZOPO2I7GSCBXS5ZZFE7TAHH6UZ3IXCVSYE3SA, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.331220212 (567h15m30.68400531s) [noise=true , address=145.79.227.17:4972 , piece_id=PMYFECZG5AFFKCRUKVKDBTGSZB7EY4RTCYMC3BIJ6ICHUAT6DKQQ, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.331462035 (567h15m30.684247133s) [noise=true , address=77.90.31.18:4807 , piece_id=DDZDZ54567IASRWONCAZ36QRULFSAUWZULGJJRZWNLCWERLIEMHQ, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.422133167 (567h15m30.774918264s) [upload=499231]: Next returned nil segment Sep 22 05:23:55.422144337 (567h15m30.774929436s) [upload=499231]: Waiting for error group managing segments... Sep 22 05:23:55.431634607 (567h15m30.784419734s) [noise=true , address=45.91.116.12:4226 , piece_id=455PMCHWJV5N6GESEP366YVGF4QS7HRT37CV45NODERPDLC5N4SA, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:23:55.432138693 (567h15m30.784923801s) [noise=true , address=77.90.31.21:4879 , piece_id=ZZ2CDZCY3YGTWZVA5NKJ7GVVPIRKYRZ6L4S36DH2CIF5FWJ7PGAQ, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:23:55.432413549 (567h15m30.785198676s) [noise=true , address=77.90.31.19:4841 , piece_id=SE5OYVNORL2NZT6N4G2C45YBJSHJJEKSXMTWGOGIJZMI7IYHEYRA, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:23:55.432647277 (567h15m30.785432385s) [noise=true , address=45.91.116.10:4081 , piece_id=LNTHUHRZVQ344JQSSJU7K26K2KNWCCNXKE4KIXOI36QFJYSUJVKA, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:23:55.433122810 (567h15m30.785907918s) [noise=true , address=77.90.31.12:4590 , piece_id=XJ45LQOCHPMUJAUZOPO2I7GSCBXS5ZZFE7TAHH6UZ3IXCVSYE3SA, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:23:55.433167022 (567h15m30.785952151s) [noise=true , address=45.91.116.13:4153 , piece_id=UVAHQBORHD4BCN7TVFWJMZJ7IHK6EQAUHPTKF4L3SJKRGZ5XCUWQ, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:23:55.433550902 (567h15m30.78633601s) [noise=true , address=77.90.31.15:4699 , piece_id=5SDUJTAZDJMLLZD4UW2IYKT6XH5PUPZESK52INKKWWJ7TCDO6WUA, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:23:55.433714501 (567h15m30.786499629s) [noise=true , address=77.90.31.21:4878 , piece_id=Q2E6VRL6B4ACWIHPY5TBOYVNC64YI6OM52N62IR36ZZDIVOPXLCA, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:23:55.434351305 (567h15m30.787136413s) [noise=true , address=77.90.31.18:4807 , piece_id=DDZDZ54567IASRWONCAZ36QRULFSAUWZULGJJRZWNLCWERLIEMHQ, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:24:12.970123192 (567h15m48.3229083s) [noise=true , address=145.79.227.14:4839 , piece_id=YNLJDUMP2MZBOO466LOXZNXE47OEJPGMZ7XZGZRODJMVZGXQWCNQ, seg_pos={0 0}, upload=499231]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:24:13.610279617 (567h15m48.963064725s) [noise=true , address=145.79.227.5:4223 , piece_id=6J3B7KY7UXL5Q54CYWOW5WTPMGCX3SIVDU6Y3KIWSIMIOY7ZFLGQ, seg_pos={0 0}, upload=499231]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:24:25.284130208 (567h16m0.636915306s) [noise=true , address=145.79.227.3:4150 , piece_id=ODGIYTQ5KSG23ABMEPF25U2V2ZPBU7TUSM5J4BUQMHHCCYUSHTZQ, seg_pos={0 0}, upload=499231]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:24:25.284192986 (567h16m0.636978094s) [noise=true , address=145.79.227.19:4078 , piece_id=27WKGKIEYCKOGF32AVABPOMR2LZAJWA6H6XDNRCJIXWYQOG7BFYA, seg_pos={0 0}, upload=499231]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:24:25.284199528 (567h16m0.636984636s) [noise=true , address=145.79.227.7:4295 , piece_id=HRPBBJQEOKX46PKKQYYQ3OSBKZRQ6FG3UF4LA3ADDRZ4RY4HU4IA, seg_pos={0 0}, upload=499231]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:24:25.284231788 (567h16m0.637016887s) [noise=true , address=145.79.227.6:4258 , piece_id=D2X5ABUMXHX2CEIPD2ZSH3XUJONB6NM54KM734EFMXZZSKI4AEEQ, seg_pos={0 0}, upload=499231]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:24:25.284246356 (567h16m0.637031464s) [noise=true , address=145.79.227.17:4972 , piece_id=PMYFECZG5AFFKCRUKVKDBTGSZB7EY4RTCYMC3BIJ6ICHUAT6DKQQ, seg_pos={0 0}, upload=499231]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:24:25.284259801 (567h16m0.637044909s) [seg_pos={0 0}, upload=499231]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 22 05:24:25.298782565 (567h16m0.651567663s) [upload=499231]: Done uploading segments... Sep 22 05:24:25.298797994 (567h16m0.651583102s) [upload=499231]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 453.358996ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........X........X..................X...X.....X....XXX.....] duration: 423.214175ms FAIL (2025-09-22 05:24:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 423.240504ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...................X...X.......X.X.......XXX.X.....X.......X] duration: 30.015399112s FAIL (2025-09-22 05:24:25 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Sep 22 05:23:55.305745495 (567h15m30.658530594s) [upload=499231]: Starting upload Sep 22 05:23:55.305810838 (567h15m30.658595936s) [upload=499231]: Done starting upload Sep 22 05:23:55.305870911 (567h15m30.658656019s) [upload=499231]: Uploading segments... Sep 22 05:23:55.306056390 (567h15m30.658841508s) [upload=499231]: Got next segment. Inline: false Sep 22 05:23:55.306324312 (567h15m30.65910942s) [upload=499231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:23:55.306330464 (567h15m30.659115562s) [upload=499231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:23:55.329363728 (567h15m30.682148826s) [seg_pos={0 0}, upload=499231]: Begin upload segment... Sep 22 05:23:55.329392612 (567h15m30.68217771s) [seg_pos={0 0}, upload=499231]: Done begin upload segment. Sep 22 05:23:55.329412229 (567h15m30.682197327s) [noise=true , address=145.79.227.19:4078 , piece_id=27WKGKIEYCKOGF32AVABPOMR2LZAJWA6H6XDNRCJIXWYQOG7BFYA, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.329495375 (567h15m30.682280483s) [noise=true , address=45.91.116.12:4226 , piece_id=455PMCHWJV5N6GESEP366YVGF4QS7HRT37CV45NODERPDLC5N4SA, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.329724985 (567h15m30.682510093s) [noise=true , address=145.79.227.7:4295 , piece_id=HRPBBJQEOKX46PKKQYYQ3OSBKZRQ6FG3UF4LA3ADDRZ4RY4HU4IA, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.329833008 (567h15m30.682618106s) [noise=true , address=145.79.227.14:4839 , piece_id=YNLJDUMP2MZBOO466LOXZNXE47OEJPGMZ7XZGZRODJMVZGXQWCNQ, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.329991756 (567h15m30.682776864s) [noise=true , address=77.90.31.21:4879 , piece_id=ZZ2CDZCY3YGTWZVA5NKJ7GVVPIRKYRZ6L4S36DH2CIF5FWJ7PGAQ, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.330098596 (567h15m30.682883704s) [noise=true , address=145.79.227.3:4150 , piece_id=ODGIYTQ5KSG23ABMEPF25U2V2ZPBU7TUSM5J4BUQMHHCCYUSHTZQ, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.330230425 (567h15m30.683015522s) [noise=true , address=45.91.116.13:4153 , piece_id=UVAHQBORHD4BCN7TVFWJMZJ7IHK6EQAUHPTKF4L3SJKRGZ5XCUWQ, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.330332817 (567h15m30.683117914s) [noise=true , address=145.79.227.5:4223 , piece_id=6J3B7KY7UXL5Q54CYWOW5WTPMGCX3SIVDU6Y3KIWSIMIOY7ZFLGQ, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.330469012 (567h15m30.68325412s) [noise=true , address=77.90.31.21:4878 , piece_id=Q2E6VRL6B4ACWIHPY5TBOYVNC64YI6OM52N62IR36ZZDIVOPXLCA, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.330554843 (567h15m30.683339941s) [noise=true , address=145.79.227.6:4258 , piece_id=D2X5ABUMXHX2CEIPD2ZSH3XUJONB6NM54KM734EFMXZZSKI4AEEQ, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.330709423 (567h15m30.683494531s) [noise=true , address=77.90.31.15:4699 , piece_id=5SDUJTAZDJMLLZD4UW2IYKT6XH5PUPZESK52INKKWWJ7TCDO6WUA, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.330767873 (567h15m30.68355297s) [noise=true , address=77.90.31.19:4841 , piece_id=SE5OYVNORL2NZT6N4G2C45YBJSHJJEKSXMTWGOGIJZMI7IYHEYRA, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.330949173 (567h15m30.683734281s) [noise=true , address=45.91.116.10:4081 , piece_id=LNTHUHRZVQ344JQSSJU7K26K2KNWCCNXKE4KIXOI36QFJYSUJVKA, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.331003133 (567h15m30.683788241s) [noise=true , address=77.90.31.12:4590 , piece_id=XJ45LQOCHPMUJAUZOPO2I7GSCBXS5ZZFE7TAHH6UZ3IXCVSYE3SA, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.331220212 (567h15m30.68400531s) [noise=true , address=145.79.227.17:4972 , piece_id=PMYFECZG5AFFKCRUKVKDBTGSZB7EY4RTCYMC3BIJ6ICHUAT6DKQQ, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.331462035 (567h15m30.684247133s) [noise=true , address=77.90.31.18:4807 , piece_id=DDZDZ54567IASRWONCAZ36QRULFSAUWZULGJJRZWNLCWERLIEMHQ, seg_pos={0 0}, upload=499231]: Uploading piece... Sep 22 05:23:55.422133167 (567h15m30.774918264s) [upload=499231]: Next returned nil segment Sep 22 05:23:55.422144337 (567h15m30.774929436s) [upload=499231]: Waiting for error group managing segments... Sep 22 05:23:55.431634607 (567h15m30.784419734s) [noise=true , address=45.91.116.12:4226 , piece_id=455PMCHWJV5N6GESEP366YVGF4QS7HRT37CV45NODERPDLC5N4SA, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:23:55.432138693 (567h15m30.784923801s) [noise=true , address=77.90.31.21:4879 , piece_id=ZZ2CDZCY3YGTWZVA5NKJ7GVVPIRKYRZ6L4S36DH2CIF5FWJ7PGAQ, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:23:55.432413549 (567h15m30.785198676s) [noise=true , address=77.90.31.19:4841 , piece_id=SE5OYVNORL2NZT6N4G2C45YBJSHJJEKSXMTWGOGIJZMI7IYHEYRA, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:23:55.432647277 (567h15m30.785432385s) [noise=true , address=45.91.116.10:4081 , piece_id=LNTHUHRZVQ344JQSSJU7K26K2KNWCCNXKE4KIXOI36QFJYSUJVKA, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:23:55.433122810 (567h15m30.785907918s) [noise=true , address=77.90.31.12:4590 , piece_id=XJ45LQOCHPMUJAUZOPO2I7GSCBXS5ZZFE7TAHH6UZ3IXCVSYE3SA, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:23:55.433167022 (567h15m30.785952151s) [noise=true , address=45.91.116.13:4153 , piece_id=UVAHQBORHD4BCN7TVFWJMZJ7IHK6EQAUHPTKF4L3SJKRGZ5XCUWQ, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:23:55.433550902 (567h15m30.78633601s) [noise=true , address=77.90.31.15:4699 , piece_id=5SDUJTAZDJMLLZD4UW2IYKT6XH5PUPZESK52INKKWWJ7TCDO6WUA, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:23:55.433714501 (567h15m30.786499629s) [noise=true , address=77.90.31.21:4878 , piece_id=Q2E6VRL6B4ACWIHPY5TBOYVNC64YI6OM52N62IR36ZZDIVOPXLCA, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:23:55.434351305 (567h15m30.787136413s) [noise=true , address=77.90.31.18:4807 , piece_id=DDZDZ54567IASRWONCAZ36QRULFSAUWZULGJJRZWNLCWERLIEMHQ, seg_pos={0 0}, upload=499231]: Done uploading piece. err: <nil> Sep 22 05:24:12.970123192 (567h15m48.3229083s) [noise=true , address=145.79.227.14:4839 , piece_id=YNLJDUMP2MZBOO466LOXZNXE47OEJPGMZ7XZGZRODJMVZGXQWCNQ, seg_pos={0 0}, upload=499231]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:24:13.610279617 (567h15m48.963064725s) [noise=true , address=145.79.227.5:4223 , piece_id=6J3B7KY7UXL5Q54CYWOW5WTPMGCX3SIVDU6Y3KIWSIMIOY7ZFLGQ, seg_pos={0 0}, upload=499231]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:24:25.284130208 (567h16m0.636915306s) [noise=true , address=145.79.227.3:4150 , piece_id=ODGIYTQ5KSG23ABMEPF25U2V2ZPBU7TUSM5J4BUQMHHCCYUSHTZQ, seg_pos={0 0}, upload=499231]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:145.79.227.3:4150): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:24:25.284192986 (567h16m0.636978094s) [noise=true , address=145.79.227.19:4078 , piece_id=27WKGKIEYCKOGF32AVABPOMR2LZAJWA6H6XDNRCJIXWYQOG7BFYA, seg_pos={0 0}, upload=499231]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:24:25.284199528 (567h16m0.636984636s) [noise=true , address=145.79.227.7:4295 , piece_id=HRPBBJQEOKX46PKKQYYQ3OSBKZRQ6FG3UF4LA3ADDRZ4RY4HU4IA, seg_pos={0 0}, upload=499231]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:24:25.284231788 (567h16m0.637016887s) [noise=true , address=145.79.227.6:4258 , piece_id=D2X5ABUMXHX2CEIPD2ZSH3XUJONB6NM54KM734EFMXZZSKI4AEEQ, seg_pos={0 0}, upload=499231]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:24:25.284246356 (567h16m0.637031464s) [noise=true , address=145.79.227.17:4972 , piece_id=PMYFECZG5AFFKCRUKVKDBTGSZB7EY4RTCYMC3BIJ6ICHUAT6DKQQ, seg_pos={0 0}, upload=499231]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:24:25.284259801 (567h16m0.637044909s) [seg_pos={0 0}, upload=499231]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 22 05:24:25.298782565 (567h16m0.651567663s) [upload=499231]: Done uploading segments... Sep 22 05:24:25.298797994 (567h16m0.651583102s) [upload=499231]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.015414351s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.925431196s FAIL (2025-09-22 05:24:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.925450493s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.........X........X..................X...X.....X....XXX....] duration: 468.858601ms FAIL (2025-09-22 05:23:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 468.879451ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.54553112s FAIL (2025-09-22 05:23:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.545554934s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X.X..X..................] duration: 484.862323ms FAIL (2025-09-22 05:23:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443356010 (566h58m39.796141139s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443765328 (566h58m39.796550456s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444090558 (566h58m39.796875686s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444188903 (566h58m39.79697403s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444577773 (566h58m39.797362901s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444677179 (566h58m39.797462307s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444962344 (566h58m39.797747472s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445216612 (566h58m39.79800174s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445329904 (566h58m39.798115033s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445688447 (566h58m39.798473585s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445754561 (566h58m39.798539689s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446085091 (566h58m39.79887022s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446552749 (566h58m39.799337917s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447022330 (566h58m39.799807459s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447446877 (566h58m39.800232005s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447878927 (566h58m39.800664056s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.448061300 (566h58m39.800846428s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.528619474 (566h58m39.881404572s) [upload=498993]: Next returned nil segment Sep 22 05:07:04.528630534 (566h58m39.881415643s) [upload=498993]: Waiting for error group managing segments... Sep 22 05:07:04.538575067 (566h58m39.891360164s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539546860 (566h58m39.892331958s) [noise=true , address=145.79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539973170 (566h58m39.892758267s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540146776 (566h58m39.892931913s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540291638 (566h58m39.893076746s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540795884 (566h58m39.893581022s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.544155669 (566h58m39.896940767s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552098143 (566h58m39.904883281s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552115576 (566h58m39.904900674s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.041097924 (566h58m40.393883022s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.506063937 (566h58m40.858849035s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1ejnWsitdrh68vSJvjR2QnRg4kva8DFRzqvvUuDkXM6zKUDWNC, address:145.79.227.11:4736): protocol: rpc: dial tcp 145.79.227.11:4736: connect: no route to host; manager closed: conn closed: op Sep 22 05:08:04.401016602 (566h59m39.75380173s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1YnhzSepDNJ6FEFq6ifLdmaGm7BZhMYs7JWbGmShbi38gSxxXL, address:145.79.227.10:4419): protocol: rpc: dial tcp 145.79.227.10:4419: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401091332 (566h59m39.753876461s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ, address:145.79.227.12:4782): protocol: rpc: dial tcp 145.79.227.12:4782: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401101101 (566h59m39.753886229s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1gakF2Cctzc6EhFScfLfBHJdcDHqMBcey4mNoToEwbtk8Pwix5, address:145.79.227.15:4890): protocol: rpc: dial tcp 145.79.227.15:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401148540 (566h59m39.753933678s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:145.79.227.14:4866): protocol: rpc: dial tcp 145.79.227.14:4866: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401183265 (566h59m39.753968403s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401203854 (566h59m39.753988992s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:145.79.227.2:4118): protocol: rpc: dial tcp 145.79.227.2:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401243117 (566h59m39.754028245s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12mRuEpX7CMxWXVdHrTi3zF3v9t3LcqE6mJKzocajEvYHM52a7T, address:145.79.227.8:4355): protocol: rpc: dial tcp 145.79.227.8:4355: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401288963 (566h59m39.754074102s) [seg_pos={0 0}, upload=498993]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Sep 22 05:08:04.415054225 (566h59m39.767839353s) [upload=498993]: Done uploading segments... Sep 22 05:08:04.415080705 (566h59m39.767865833s) [upload=498993]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 484.897881ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....................X...X.......X.X.......XXX.X.....X.......] duration: 404.520305ms FAIL (2025-09-22 05:23:01 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:17:57.648033125 (567h9m33.000818253s) [upload=499148]: Starting upload Sep 22 05:17:57.648114798 (567h9m33.000899926s) [upload=499148]: Done starting upload Sep 22 05:17:57.648176143 (567h9m33.000961261s) [upload=499148]: Uploading segments... Sep 22 05:17:57.648452281 (567h9m33.001237409s) [upload=499148]: Got next segment. Inline: false Sep 22 05:17:57.648494821 (567h9m33.001279949s) [upload=499148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:17:57.648499850 (567h9m33.001284979s) [upload=499148]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:17:57.670664705 (567h9m33.023449833s) [seg_pos={0 0}, upload=499148]: Begin upload segment... Sep 22 05:17:57.670704479 (567h9m33.023489608s) [seg_pos={0 0}, upload=499148]: Done begin upload segment. Sep 22 05:17:57.670739555 (567h9m33.023524693s) [noise=true , address=45.91.116.12:4225 , piece_id=NYCEKKNFDFVIYUEJEYJ6TH7ZQWZILNECJUTYZQYTH3H75IG2VRJA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671214056 (567h9m33.023999184s) [noise=true , address=77.90.31.13:4628 , piece_id=ILDBZSYPQSFOJMQJIJ3YY6ZAEGGALRP6NCBFB77BUUZF7EHWDLOQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671585262 (567h9m33.0243704s) [noise=true , address=45.91.116.5:4299 , piece_id=AIADMFRASUJDIGECRYLT3J3VSCY73HLWLKQGSWIQ27KMNH7ZYKZA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671997806 (567h9m33.024782934s) [noise=true , address=77.90.31.21:4879 , piece_id=Z7GY3H6OT4BHOABHLY7RXNEEE2LRWQCY2EVEPONLFW6K2YHTBCSA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.672507884 (567h9m33.025293172s) [noise=true , address=77.90.31.12:4592 , piece_id=FOTIXM3HUVN5RSNCONJMDYW2A2FQKJGIFXFH3HPRQMCYDIZ3D2JA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673048238 (567h9m33.025833366s) [noise=true , address=45.91.116.12:4227 , piece_id=W32WUPFKSPWRWBCL7JADKWIU7EQODOI27T7KTPN7WCVCZCUQ4OXQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673455823 (567h9m33.02624096s) [noise=true , address=45.91.116.4:4937 , piece_id=S4AQXWAPQ6CR42ATHGKIEN36SVHBWNMLBGULGMORWHRCPC2CA4AA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673895858 (567h9m33.026680996s) [noise=true , address=45.91.116.13:4155 , piece_id=WQYODY6JA56UQYQHVIU4N4OAEVDVUSNM7WEAS3KLA22JXZ3NS73A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.674287894 (567h9m33.027073022s) [noise=true , address=45.91.116.14:4190 , piece_id=QZUCXRPJXFHAQD372QMII7WLQ3EXRMB5TAXHLOFPA6JJJCJSEQ2A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.674646156 (567h9m33.027431284s) [noise=true , address=45.91.116.8:4047 , piece_id=JZNCWA7323AGW3MBN2AFPYOFTUTZHS3ULF23ZKZVIEHVDCVUXMXA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675012384 (567h9m33.027797512s) [noise=true , address=145.79.227.16:4909 , piece_id=BKRUZJZ7CVO6XTOS7ASPCRHC3ZHNTZRW62FZDZJ2VKUUFHIAYA4Q, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675382799 (567h9m33.028167936s) [noise=true , address=45.91.116.14:4189 , piece_id=4ZNEJODQNSUDF3AJR3GFEAABDOYPYVJBW4TYH7IXSHOAMLYEU25Q, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675735730 (567h9m33.028520859s) [noise=true , address=145.79.227.17:4972 , piece_id=YBWZPNXLC7DDYVSQ5UY3JI7FYI3HKRVRRAECT7B4SZ3DQN32GSQA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676142213 (567h9m33.028927352s) [noise=true , address=145.79.227.15:4874 , piece_id=IKJBG6PIHIA6RGACRBBZMEHG5TVWYORKJOUKOPU2C7PXTV2XOJ7A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676508621 (567h9m33.029293749s) [noise=true , address=77.90.31.17:4770 , piece_id=7JWSWRVKROEAHCFZD2CVFRYFIMBYQRS2IH72NQYADFPDVDZ5LDOQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676883124 (567h9m33.029668252s) [noise=true , address=145.79.227.13:4803 , piece_id=RFE4LY72VJIEIB3ZRUO2JF64ME4KHI6MI43Z4BGPW2LBBQ3WGWTA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.787459380 (567h9m33.140244498s) [upload=499148]: Next returned nil segment Sep 22 05:17:57.787474078 (567h9m33.140259205s) [upload=499148]: Waiting for error group managing segments... Sep 22 05:17:57.797424119 (567h9m33.150209227s) [noise=true , address=77.90.31.12:4592 , piece_id=FOTIXM3HUVN5RSNCONJMDYW2A2FQKJGIFXFH3HPRQMCYDIZ3D2JA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.797935569 (567h9m33.150720677s) [noise=true , address=45.91.116.8:4047 , piece_id=JZNCWA7323AGW3MBN2AFPYOFTUTZHS3ULF23ZKZVIEHVDCVUXMXA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798273803 (567h9m33.151058901s) [noise=true , address=45.91.116.12:4227 , piece_id=W32WUPFKSPWRWBCL7JADKWIU7EQODOI27T7KTPN7WCVCZCUQ4OXQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798418767 (567h9m33.151203875s) [noise=true , address=45.91.116.13:4155 , piece_id=WQYODY6JA56UQYQHVIU4N4OAEVDVUSNM7WEAS3KLA22JXZ3NS73A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798591931 (567h9m33.15137704s) [noise=true , address=45.91.116.5:4299 , piece_id=AIADMFRASUJDIGECRYLT3J3VSCY73HLWLKQGSWIQ27KMNH7ZYKZA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798814057 (567h9m33.151599156s) [noise=true , address=77.90.31.13:4628 , piece_id=ILDBZSYPQSFOJMQJIJ3YY6ZAEGGALRP6NCBFB77BUUZF7EHWDLOQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798903976 (567h9m33.151689084s) [noise=true , address=45.91.116.14:4189 , piece_id=4ZNEJODQNSUDF3AJR3GFEAABDOYPYVJBW4TYH7IXSHOAMLYEU25Q, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799239796 (567h9m33.152024904s) [noise=true , address=45.91.116.4:4937 , piece_id=S4AQXWAPQ6CR42ATHGKIEN36SVHBWNMLBGULGMORWHRCPC2CA4AA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799291223 (567h9m33.152076321s) [noise=true , address=45.91.116.12:4225 , piece_id=NYCEKKNFDFVIYUEJEYJ6TH7ZQWZILNECJUTYZQYTH3H75IG2VRJA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799717784 (567h9m33.152502882s) [noise=true , address=45.91.116.14:4190 , piece_id=QZUCXRPJXFHAQD372QMII7WLQ3EXRMB5TAXHLOFPA6JJJCJSEQ2A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.800251245 (567h9m33.153036353s) [noise=true , address=77.90.31.17:4770 , piece_id=7JWSWRVKROEAHCFZD2CVFRYFIMBYQRS2IH72NQYADFPDVDZ5LDOQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.800580202 (567h9m33.15336531s) [noise=true , address=77.90.31.21:4879 , piece_id=Z7GY3H6OT4BHOABHLY7RXNEEE2LRWQCY2EVEPONLFW6K2YHTBCSA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:18:27.632317337 (567h10m2.985102445s) [noise=true , address=145.79.227.17:4972 , piece_id=YBWZPNXLC7DDYVSQ5UY3JI7FYI3HKRVRRAECT7B4SZ3DQN32GSQA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632381407 (567h10m2.985166515s) [noise=true , address=145.79.227.13:4803 , piece_id=RFE4LY72VJIEIB3ZRUO2JF64ME4KHI6MI43Z4BGPW2LBBQ3WGWTA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632413378 (567h10m2.985198476s) [noise=true , address=145.79.227.16:4909 , piece_id=BKRUZJZ7CVO6XTOS7ASPCRHC3ZHNTZRW62FZDZJ2VKUUFHIAYA4Q, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:27.632470525 (567h10m2.985255623s) [noise=true , address=145.79.227.15:4874 , piece_id=IKJBG6PIHIA6RGACRBBZMEHG5TVWYORKJOUKOPU2C7PXTV2XOJ7A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632487837 (567h10m2.985272945s) [seg_pos={0 0}, upload=499148]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:18:27.648303797 (567h10m3.001088895s) [upload=499148]: Done uploading segments... Sep 22 05:18:27.648323044 (567h10m3.001108141s) [upload=499148]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 404.544641ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XX.........X........X..................X...X.....X....XXX...] duration: 427.26658ms FAIL (2025-09-22 05:22:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 427.283622ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....................X...X.......X.X.......XXX.X.....X......] duration: 364.089283ms FAIL (2025-09-22 05:22:42 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:17:57.648033125 (567h9m33.000818253s) [upload=499148]: Starting upload Sep 22 05:17:57.648114798 (567h9m33.000899926s) [upload=499148]: Done starting upload Sep 22 05:17:57.648176143 (567h9m33.000961261s) [upload=499148]: Uploading segments... Sep 22 05:17:57.648452281 (567h9m33.001237409s) [upload=499148]: Got next segment. Inline: false Sep 22 05:17:57.648494821 (567h9m33.001279949s) [upload=499148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:17:57.648499850 (567h9m33.001284979s) [upload=499148]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:17:57.670664705 (567h9m33.023449833s) [seg_pos={0 0}, upload=499148]: Begin upload segment... Sep 22 05:17:57.670704479 (567h9m33.023489608s) [seg_pos={0 0}, upload=499148]: Done begin upload segment. Sep 22 05:17:57.670739555 (567h9m33.023524693s) [noise=true , address=45.91.116.12:4225 , piece_id=NYCEKKNFDFVIYUEJEYJ6TH7ZQWZILNECJUTYZQYTH3H75IG2VRJA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671214056 (567h9m33.023999184s) [noise=true , address=77.90.31.13:4628 , piece_id=ILDBZSYPQSFOJMQJIJ3YY6ZAEGGALRP6NCBFB77BUUZF7EHWDLOQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671585262 (567h9m33.0243704s) [noise=true , address=45.91.116.5:4299 , piece_id=AIADMFRASUJDIGECRYLT3J3VSCY73HLWLKQGSWIQ27KMNH7ZYKZA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671997806 (567h9m33.024782934s) [noise=true , address=77.90.31.21:4879 , piece_id=Z7GY3H6OT4BHOABHLY7RXNEEE2LRWQCY2EVEPONLFW6K2YHTBCSA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.672507884 (567h9m33.025293172s) [noise=true , address=77.90.31.12:4592 , piece_id=FOTIXM3HUVN5RSNCONJMDYW2A2FQKJGIFXFH3HPRQMCYDIZ3D2JA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673048238 (567h9m33.025833366s) [noise=true , address=45.91.116.12:4227 , piece_id=W32WUPFKSPWRWBCL7JADKWIU7EQODOI27T7KTPN7WCVCZCUQ4OXQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673455823 (567h9m33.02624096s) [noise=true , address=45.91.116.4:4937 , piece_id=S4AQXWAPQ6CR42ATHGKIEN36SVHBWNMLBGULGMORWHRCPC2CA4AA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673895858 (567h9m33.026680996s) [noise=true , address=45.91.116.13:4155 , piece_id=WQYODY6JA56UQYQHVIU4N4OAEVDVUSNM7WEAS3KLA22JXZ3NS73A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.674287894 (567h9m33.027073022s) [noise=true , address=45.91.116.14:4190 , piece_id=QZUCXRPJXFHAQD372QMII7WLQ3EXRMB5TAXHLOFPA6JJJCJSEQ2A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.674646156 (567h9m33.027431284s) [noise=true , address=45.91.116.8:4047 , piece_id=JZNCWA7323AGW3MBN2AFPYOFTUTZHS3ULF23ZKZVIEHVDCVUXMXA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675012384 (567h9m33.027797512s) [noise=true , address=145.79.227.16:4909 , piece_id=BKRUZJZ7CVO6XTOS7ASPCRHC3ZHNTZRW62FZDZJ2VKUUFHIAYA4Q, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675382799 (567h9m33.028167936s) [noise=true , address=45.91.116.14:4189 , piece_id=4ZNEJODQNSUDF3AJR3GFEAABDOYPYVJBW4TYH7IXSHOAMLYEU25Q, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675735730 (567h9m33.028520859s) [noise=true , address=145.79.227.17:4972 , piece_id=YBWZPNXLC7DDYVSQ5UY3JI7FYI3HKRVRRAECT7B4SZ3DQN32GSQA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676142213 (567h9m33.028927352s) [noise=true , address=145.79.227.15:4874 , piece_id=IKJBG6PIHIA6RGACRBBZMEHG5TVWYORKJOUKOPU2C7PXTV2XOJ7A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676508621 (567h9m33.029293749s) [noise=true , address=77.90.31.17:4770 , piece_id=7JWSWRVKROEAHCFZD2CVFRYFIMBYQRS2IH72NQYADFPDVDZ5LDOQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676883124 (567h9m33.029668252s) [noise=true , address=145.79.227.13:4803 , piece_id=RFE4LY72VJIEIB3ZRUO2JF64ME4KHI6MI43Z4BGPW2LBBQ3WGWTA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.787459380 (567h9m33.140244498s) [upload=499148]: Next returned nil segment Sep 22 05:17:57.787474078 (567h9m33.140259205s) [upload=499148]: Waiting for error group managing segments... Sep 22 05:17:57.797424119 (567h9m33.150209227s) [noise=true , address=77.90.31.12:4592 , piece_id=FOTIXM3HUVN5RSNCONJMDYW2A2FQKJGIFXFH3HPRQMCYDIZ3D2JA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.797935569 (567h9m33.150720677s) [noise=true , address=45.91.116.8:4047 , piece_id=JZNCWA7323AGW3MBN2AFPYOFTUTZHS3ULF23ZKZVIEHVDCVUXMXA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798273803 (567h9m33.151058901s) [noise=true , address=45.91.116.12:4227 , piece_id=W32WUPFKSPWRWBCL7JADKWIU7EQODOI27T7KTPN7WCVCZCUQ4OXQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798418767 (567h9m33.151203875s) [noise=true , address=45.91.116.13:4155 , piece_id=WQYODY6JA56UQYQHVIU4N4OAEVDVUSNM7WEAS3KLA22JXZ3NS73A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798591931 (567h9m33.15137704s) [noise=true , address=45.91.116.5:4299 , piece_id=AIADMFRASUJDIGECRYLT3J3VSCY73HLWLKQGSWIQ27KMNH7ZYKZA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798814057 (567h9m33.151599156s) [noise=true , address=77.90.31.13:4628 , piece_id=ILDBZSYPQSFOJMQJIJ3YY6ZAEGGALRP6NCBFB77BUUZF7EHWDLOQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798903976 (567h9m33.151689084s) [noise=true , address=45.91.116.14:4189 , piece_id=4ZNEJODQNSUDF3AJR3GFEAABDOYPYVJBW4TYH7IXSHOAMLYEU25Q, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799239796 (567h9m33.152024904s) [noise=true , address=45.91.116.4:4937 , piece_id=S4AQXWAPQ6CR42ATHGKIEN36SVHBWNMLBGULGMORWHRCPC2CA4AA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799291223 (567h9m33.152076321s) [noise=true , address=45.91.116.12:4225 , piece_id=NYCEKKNFDFVIYUEJEYJ6TH7ZQWZILNECJUTYZQYTH3H75IG2VRJA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799717784 (567h9m33.152502882s) [noise=true , address=45.91.116.14:4190 , piece_id=QZUCXRPJXFHAQD372QMII7WLQ3EXRMB5TAXHLOFPA6JJJCJSEQ2A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.800251245 (567h9m33.153036353s) [noise=true , address=77.90.31.17:4770 , piece_id=7JWSWRVKROEAHCFZD2CVFRYFIMBYQRS2IH72NQYADFPDVDZ5LDOQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.800580202 (567h9m33.15336531s) [noise=true , address=77.90.31.21:4879 , piece_id=Z7GY3H6OT4BHOABHLY7RXNEEE2LRWQCY2EVEPONLFW6K2YHTBCSA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:18:27.632317337 (567h10m2.985102445s) [noise=true , address=145.79.227.17:4972 , piece_id=YBWZPNXLC7DDYVSQ5UY3JI7FYI3HKRVRRAECT7B4SZ3DQN32GSQA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632381407 (567h10m2.985166515s) [noise=true , address=145.79.227.13:4803 , piece_id=RFE4LY72VJIEIB3ZRUO2JF64ME4KHI6MI43Z4BGPW2LBBQ3WGWTA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632413378 (567h10m2.985198476s) [noise=true , address=145.79.227.16:4909 , piece_id=BKRUZJZ7CVO6XTOS7ASPCRHC3ZHNTZRW62FZDZJ2VKUUFHIAYA4Q, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:27.632470525 (567h10m2.985255623s) [noise=true , address=145.79.227.15:4874 , piece_id=IKJBG6PIHIA6RGACRBBZMEHG5TVWYORKJOUKOPU2C7PXTV2XOJ7A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632487837 (567h10m2.985272945s) [seg_pos={0 0}, upload=499148]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:18:27.648303797 (567h10m3.001088895s) [upload=499148]: Done uploading segments... Sep 22 05:18:27.648323044 (567h10m3.001108141s) [upload=499148]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 364.14095ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X.X..X.................] duration: 453.815882ms FAIL (2025-09-22 05:22:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443356010 (566h58m39.796141139s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443765328 (566h58m39.796550456s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444090558 (566h58m39.796875686s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444188903 (566h58m39.79697403s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444577773 (566h58m39.797362901s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444677179 (566h58m39.797462307s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444962344 (566h58m39.797747472s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445216612 (566h58m39.79800174s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445329904 (566h58m39.798115033s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445688447 (566h58m39.798473585s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445754561 (566h58m39.798539689s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446085091 (566h58m39.79887022s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446552749 (566h58m39.799337917s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447022330 (566h58m39.799807459s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447446877 (566h58m39.800232005s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447878927 (566h58m39.800664056s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.448061300 (566h58m39.800846428s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.528619474 (566h58m39.881404572s) [upload=498993]: Next returned nil segment Sep 22 05:07:04.528630534 (566h58m39.881415643s) [upload=498993]: Waiting for error group managing segments... Sep 22 05:07:04.538575067 (566h58m39.891360164s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539546860 (566h58m39.892331958s) [noise=true , address=145.79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539973170 (566h58m39.892758267s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540146776 (566h58m39.892931913s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540291638 (566h58m39.893076746s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540795884 (566h58m39.893581022s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.544155669 (566h58m39.896940767s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552098143 (566h58m39.904883281s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552115576 (566h58m39.904900674s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.041097924 (566h58m40.393883022s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.506063937 (566h58m40.858849035s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1ejnWsitdrh68vSJvjR2QnRg4kva8DFRzqvvUuDkXM6zKUDWNC, address:145.79.227.11:4736): protocol: rpc: dial tcp 145.79.227.11:4736: connect: no route to host; manager closed: conn closed: op Sep 22 05:08:04.401016602 (566h59m39.75380173s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1YnhzSepDNJ6FEFq6ifLdmaGm7BZhMYs7JWbGmShbi38gSxxXL, address:145.79.227.10:4419): protocol: rpc: dial tcp 145.79.227.10:4419: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401091332 (566h59m39.753876461s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ, address:145.79.227.12:4782): protocol: rpc: dial tcp 145.79.227.12:4782: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401101101 (566h59m39.753886229s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1gakF2Cctzc6EhFScfLfBHJdcDHqMBcey4mNoToEwbtk8Pwix5, address:145.79.227.15:4890): protocol: rpc: dial tcp 145.79.227.15:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401148540 (566h59m39.753933678s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:145.79.227.14:4866): protocol: rpc: dial tcp 145.79.227.14:4866: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401183265 (566h59m39.753968403s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401203854 (566h59m39.753988992s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:145.79.227.2:4118): protocol: rpc: dial tcp 145.79.227.2:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401243117 (566h59m39.754028245s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12mRuEpX7CMxWXVdHrTi3zF3v9t3LcqE6mJKzocajEvYHM52a7T, address:145.79.227.8:4355): protocol: rpc: dial tcp 145.79.227.8:4355: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401288963 (566h59m39.754074102s) [seg_pos={0 0}, upload=498993]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Sep 22 05:08:04.415054225 (566h59m39.767839353s) [upload=498993]: Done uploading segments... Sep 22 05:08:04.415080705 (566h59m39.767865833s) [upload=498993]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 453.833896ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.412289045s FAIL (2025-09-22 05:22:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.412318289s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......................X...X.......X.X.......XXX.X.....X.....] duration: 458.249913ms FAIL (2025-09-22 05:22:16 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:17:57.648033125 (567h9m33.000818253s) [upload=499148]: Starting upload Sep 22 05:17:57.648114798 (567h9m33.000899926s) [upload=499148]: Done starting upload Sep 22 05:17:57.648176143 (567h9m33.000961261s) [upload=499148]: Uploading segments... Sep 22 05:17:57.648452281 (567h9m33.001237409s) [upload=499148]: Got next segment. Inline: false Sep 22 05:17:57.648494821 (567h9m33.001279949s) [upload=499148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:17:57.648499850 (567h9m33.001284979s) [upload=499148]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:17:57.670664705 (567h9m33.023449833s) [seg_pos={0 0}, upload=499148]: Begin upload segment... Sep 22 05:17:57.670704479 (567h9m33.023489608s) [seg_pos={0 0}, upload=499148]: Done begin upload segment. Sep 22 05:17:57.670739555 (567h9m33.023524693s) [noise=true , address=45.91.116.12:4225 , piece_id=NYCEKKNFDFVIYUEJEYJ6TH7ZQWZILNECJUTYZQYTH3H75IG2VRJA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671214056 (567h9m33.023999184s) [noise=true , address=77.90.31.13:4628 , piece_id=ILDBZSYPQSFOJMQJIJ3YY6ZAEGGALRP6NCBFB77BUUZF7EHWDLOQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671585262 (567h9m33.0243704s) [noise=true , address=45.91.116.5:4299 , piece_id=AIADMFRASUJDIGECRYLT3J3VSCY73HLWLKQGSWIQ27KMNH7ZYKZA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671997806 (567h9m33.024782934s) [noise=true , address=77.90.31.21:4879 , piece_id=Z7GY3H6OT4BHOABHLY7RXNEEE2LRWQCY2EVEPONLFW6K2YHTBCSA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.672507884 (567h9m33.025293172s) [noise=true , address=77.90.31.12:4592 , piece_id=FOTIXM3HUVN5RSNCONJMDYW2A2FQKJGIFXFH3HPRQMCYDIZ3D2JA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673048238 (567h9m33.025833366s) [noise=true , address=45.91.116.12:4227 , piece_id=W32WUPFKSPWRWBCL7JADKWIU7EQODOI27T7KTPN7WCVCZCUQ4OXQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673455823 (567h9m33.02624096s) [noise=true , address=45.91.116.4:4937 , piece_id=S4AQXWAPQ6CR42ATHGKIEN36SVHBWNMLBGULGMORWHRCPC2CA4AA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673895858 (567h9m33.026680996s) [noise=true , address=45.91.116.13:4155 , piece_id=WQYODY6JA56UQYQHVIU4N4OAEVDVUSNM7WEAS3KLA22JXZ3NS73A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.674287894 (567h9m33.027073022s) [noise=true , address=45.91.116.14:4190 , piece_id=QZUCXRPJXFHAQD372QMII7WLQ3EXRMB5TAXHLOFPA6JJJCJSEQ2A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.674646156 (567h9m33.027431284s) [noise=true , address=45.91.116.8:4047 , piece_id=JZNCWA7323AGW3MBN2AFPYOFTUTZHS3ULF23ZKZVIEHVDCVUXMXA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675012384 (567h9m33.027797512s) [noise=true , address=145.79.227.16:4909 , piece_id=BKRUZJZ7CVO6XTOS7ASPCRHC3ZHNTZRW62FZDZJ2VKUUFHIAYA4Q, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675382799 (567h9m33.028167936s) [noise=true , address=45.91.116.14:4189 , piece_id=4ZNEJODQNSUDF3AJR3GFEAABDOYPYVJBW4TYH7IXSHOAMLYEU25Q, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675735730 (567h9m33.028520859s) [noise=true , address=145.79.227.17:4972 , piece_id=YBWZPNXLC7DDYVSQ5UY3JI7FYI3HKRVRRAECT7B4SZ3DQN32GSQA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676142213 (567h9m33.028927352s) [noise=true , address=145.79.227.15:4874 , piece_id=IKJBG6PIHIA6RGACRBBZMEHG5TVWYORKJOUKOPU2C7PXTV2XOJ7A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676508621 (567h9m33.029293749s) [noise=true , address=77.90.31.17:4770 , piece_id=7JWSWRVKROEAHCFZD2CVFRYFIMBYQRS2IH72NQYADFPDVDZ5LDOQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676883124 (567h9m33.029668252s) [noise=true , address=145.79.227.13:4803 , piece_id=RFE4LY72VJIEIB3ZRUO2JF64ME4KHI6MI43Z4BGPW2LBBQ3WGWTA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.787459380 (567h9m33.140244498s) [upload=499148]: Next returned nil segment Sep 22 05:17:57.787474078 (567h9m33.140259205s) [upload=499148]: Waiting for error group managing segments... Sep 22 05:17:57.797424119 (567h9m33.150209227s) [noise=true , address=77.90.31.12:4592 , piece_id=FOTIXM3HUVN5RSNCONJMDYW2A2FQKJGIFXFH3HPRQMCYDIZ3D2JA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.797935569 (567h9m33.150720677s) [noise=true , address=45.91.116.8:4047 , piece_id=JZNCWA7323AGW3MBN2AFPYOFTUTZHS3ULF23ZKZVIEHVDCVUXMXA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798273803 (567h9m33.151058901s) [noise=true , address=45.91.116.12:4227 , piece_id=W32WUPFKSPWRWBCL7JADKWIU7EQODOI27T7KTPN7WCVCZCUQ4OXQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798418767 (567h9m33.151203875s) [noise=true , address=45.91.116.13:4155 , piece_id=WQYODY6JA56UQYQHVIU4N4OAEVDVUSNM7WEAS3KLA22JXZ3NS73A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798591931 (567h9m33.15137704s) [noise=true , address=45.91.116.5:4299 , piece_id=AIADMFRASUJDIGECRYLT3J3VSCY73HLWLKQGSWIQ27KMNH7ZYKZA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798814057 (567h9m33.151599156s) [noise=true , address=77.90.31.13:4628 , piece_id=ILDBZSYPQSFOJMQJIJ3YY6ZAEGGALRP6NCBFB77BUUZF7EHWDLOQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798903976 (567h9m33.151689084s) [noise=true , address=45.91.116.14:4189 , piece_id=4ZNEJODQNSUDF3AJR3GFEAABDOYPYVJBW4TYH7IXSHOAMLYEU25Q, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799239796 (567h9m33.152024904s) [noise=true , address=45.91.116.4:4937 , piece_id=S4AQXWAPQ6CR42ATHGKIEN36SVHBWNMLBGULGMORWHRCPC2CA4AA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799291223 (567h9m33.152076321s) [noise=true , address=45.91.116.12:4225 , piece_id=NYCEKKNFDFVIYUEJEYJ6TH7ZQWZILNECJUTYZQYTH3H75IG2VRJA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799717784 (567h9m33.152502882s) [noise=true , address=45.91.116.14:4190 , piece_id=QZUCXRPJXFHAQD372QMII7WLQ3EXRMB5TAXHLOFPA6JJJCJSEQ2A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.800251245 (567h9m33.153036353s) [noise=true , address=77.90.31.17:4770 , piece_id=7JWSWRVKROEAHCFZD2CVFRYFIMBYQRS2IH72NQYADFPDVDZ5LDOQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.800580202 (567h9m33.15336531s) [noise=true , address=77.90.31.21:4879 , piece_id=Z7GY3H6OT4BHOABHLY7RXNEEE2LRWQCY2EVEPONLFW6K2YHTBCSA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:18:27.632317337 (567h10m2.985102445s) [noise=true , address=145.79.227.17:4972 , piece_id=YBWZPNXLC7DDYVSQ5UY3JI7FYI3HKRVRRAECT7B4SZ3DQN32GSQA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632381407 (567h10m2.985166515s) [noise=true , address=145.79.227.13:4803 , piece_id=RFE4LY72VJIEIB3ZRUO2JF64ME4KHI6MI43Z4BGPW2LBBQ3WGWTA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632413378 (567h10m2.985198476s) [noise=true , address=145.79.227.16:4909 , piece_id=BKRUZJZ7CVO6XTOS7ASPCRHC3ZHNTZRW62FZDZJ2VKUUFHIAYA4Q, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:27.632470525 (567h10m2.985255623s) [noise=true , address=145.79.227.15:4874 , piece_id=IKJBG6PIHIA6RGACRBBZMEHG5TVWYORKJOUKOPU2C7PXTV2XOJ7A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632487837 (567h10m2.985272945s) [seg_pos={0 0}, upload=499148]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:18:27.648303797 (567h10m3.001088895s) [upload=499148]: Done uploading segments... Sep 22 05:18:27.648323044 (567h10m3.001108141s) [upload=499148]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 458.272715ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X.X..X................] duration: 423.065266ms FAIL (2025-09-22 05:22:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443356010 (566h58m39.796141139s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443765328 (566h58m39.796550456s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444090558 (566h58m39.796875686s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444188903 (566h58m39.79697403s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444577773 (566h58m39.797362901s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444677179 (566h58m39.797462307s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444962344 (566h58m39.797747472s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445216612 (566h58m39.79800174s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445329904 (566h58m39.798115033s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445688447 (566h58m39.798473585s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445754561 (566h58m39.798539689s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446085091 (566h58m39.79887022s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446552749 (566h58m39.799337917s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447022330 (566h58m39.799807459s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447446877 (566h58m39.800232005s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447878927 (566h58m39.800664056s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.448061300 (566h58m39.800846428s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.528619474 (566h58m39.881404572s) [upload=498993]: Next returned nil segment Sep 22 05:07:04.528630534 (566h58m39.881415643s) [upload=498993]: Waiting for error group managing segments... Sep 22 05:07:04.538575067 (566h58m39.891360164s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539546860 (566h58m39.892331958s) [noise=true , address=145.79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539973170 (566h58m39.892758267s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540146776 (566h58m39.892931913s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540291638 (566h58m39.893076746s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540795884 (566h58m39.893581022s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.544155669 (566h58m39.896940767s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552098143 (566h58m39.904883281s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552115576 (566h58m39.904900674s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.041097924 (566h58m40.393883022s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.506063937 (566h58m40.858849035s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1ejnWsitdrh68vSJvjR2QnRg4kva8DFRzqvvUuDkXM6zKUDWNC, address:145.79.227.11:4736): protocol: rpc: dial tcp 145.79.227.11:4736: connect: no route to host; manager closed: conn closed: op Sep 22 05:08:04.401016602 (566h59m39.75380173s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1YnhzSepDNJ6FEFq6ifLdmaGm7BZhMYs7JWbGmShbi38gSxxXL, address:145.79.227.10:4419): protocol: rpc: dial tcp 145.79.227.10:4419: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401091332 (566h59m39.753876461s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ, address:145.79.227.12:4782): protocol: rpc: dial tcp 145.79.227.12:4782: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401101101 (566h59m39.753886229s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1gakF2Cctzc6EhFScfLfBHJdcDHqMBcey4mNoToEwbtk8Pwix5, address:145.79.227.15:4890): protocol: rpc: dial tcp 145.79.227.15:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401148540 (566h59m39.753933678s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:145.79.227.14:4866): protocol: rpc: dial tcp 145.79.227.14:4866: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401183265 (566h59m39.753968403s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401203854 (566h59m39.753988992s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:145.79.227.2:4118): protocol: rpc: dial tcp 145.79.227.2:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401243117 (566h59m39.754028245s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12mRuEpX7CMxWXVdHrTi3zF3v9t3LcqE6mJKzocajEvYHM52a7T, address:145.79.227.8:4355): protocol: rpc: dial tcp 145.79.227.8:4355: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401288963 (566h59m39.754074102s) [seg_pos={0 0}, upload=498993]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Sep 22 05:08:04.415054225 (566h59m39.767839353s) [upload=498993]: Done uploading segments... Sep 22 05:08:04.415080705 (566h59m39.767865833s) [upload=498993]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 423.090213ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXX.........X........X..................X...X.....X....XXX..] duration: 468.338787ms FAIL (2025-09-22 05:21:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 468.357762ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.577668666s FAIL (2025-09-22 05:21:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.577689746s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......................X...X.......X.X.......XXX.X.....X....] duration: 527.679843ms FAIL (2025-09-22 05:21:11 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:17:57.648033125 (567h9m33.000818253s) [upload=499148]: Starting upload Sep 22 05:17:57.648114798 (567h9m33.000899926s) [upload=499148]: Done starting upload Sep 22 05:17:57.648176143 (567h9m33.000961261s) [upload=499148]: Uploading segments... Sep 22 05:17:57.648452281 (567h9m33.001237409s) [upload=499148]: Got next segment. Inline: false Sep 22 05:17:57.648494821 (567h9m33.001279949s) [upload=499148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:17:57.648499850 (567h9m33.001284979s) [upload=499148]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:17:57.670664705 (567h9m33.023449833s) [seg_pos={0 0}, upload=499148]: Begin upload segment... Sep 22 05:17:57.670704479 (567h9m33.023489608s) [seg_pos={0 0}, upload=499148]: Done begin upload segment. Sep 22 05:17:57.670739555 (567h9m33.023524693s) [noise=true , address=45.91.116.12:4225 , piece_id=NYCEKKNFDFVIYUEJEYJ6TH7ZQWZILNECJUTYZQYTH3H75IG2VRJA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671214056 (567h9m33.023999184s) [noise=true , address=77.90.31.13:4628 , piece_id=ILDBZSYPQSFOJMQJIJ3YY6ZAEGGALRP6NCBFB77BUUZF7EHWDLOQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671585262 (567h9m33.0243704s) [noise=true , address=45.91.116.5:4299 , piece_id=AIADMFRASUJDIGECRYLT3J3VSCY73HLWLKQGSWIQ27KMNH7ZYKZA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671997806 (567h9m33.024782934s) [noise=true , address=77.90.31.21:4879 , piece_id=Z7GY3H6OT4BHOABHLY7RXNEEE2LRWQCY2EVEPONLFW6K2YHTBCSA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.672507884 (567h9m33.025293172s) [noise=true , address=77.90.31.12:4592 , piece_id=FOTIXM3HUVN5RSNCONJMDYW2A2FQKJGIFXFH3HPRQMCYDIZ3D2JA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673048238 (567h9m33.025833366s) [noise=true , address=45.91.116.12:4227 , piece_id=W32WUPFKSPWRWBCL7JADKWIU7EQODOI27T7KTPN7WCVCZCUQ4OXQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673455823 (567h9m33.02624096s) [noise=true , address=45.91.116.4:4937 , piece_id=S4AQXWAPQ6CR42ATHGKIEN36SVHBWNMLBGULGMORWHRCPC2CA4AA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673895858 (567h9m33.026680996s) [noise=true , address=45.91.116.13:4155 , piece_id=WQYODY6JA56UQYQHVIU4N4OAEVDVUSNM7WEAS3KLA22JXZ3NS73A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.674287894 (567h9m33.027073022s) [noise=true , address=45.91.116.14:4190 , piece_id=QZUCXRPJXFHAQD372QMII7WLQ3EXRMB5TAXHLOFPA6JJJCJSEQ2A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.674646156 (567h9m33.027431284s) [noise=true , address=45.91.116.8:4047 , piece_id=JZNCWA7323AGW3MBN2AFPYOFTUTZHS3ULF23ZKZVIEHVDCVUXMXA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675012384 (567h9m33.027797512s) [noise=true , address=145.79.227.16:4909 , piece_id=BKRUZJZ7CVO6XTOS7ASPCRHC3ZHNTZRW62FZDZJ2VKUUFHIAYA4Q, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675382799 (567h9m33.028167936s) [noise=true , address=45.91.116.14:4189 , piece_id=4ZNEJODQNSUDF3AJR3GFEAABDOYPYVJBW4TYH7IXSHOAMLYEU25Q, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675735730 (567h9m33.028520859s) [noise=true , address=145.79.227.17:4972 , piece_id=YBWZPNXLC7DDYVSQ5UY3JI7FYI3HKRVRRAECT7B4SZ3DQN32GSQA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676142213 (567h9m33.028927352s) [noise=true , address=145.79.227.15:4874 , piece_id=IKJBG6PIHIA6RGACRBBZMEHG5TVWYORKJOUKOPU2C7PXTV2XOJ7A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676508621 (567h9m33.029293749s) [noise=true , address=77.90.31.17:4770 , piece_id=7JWSWRVKROEAHCFZD2CVFRYFIMBYQRS2IH72NQYADFPDVDZ5LDOQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676883124 (567h9m33.029668252s) [noise=true , address=145.79.227.13:4803 , piece_id=RFE4LY72VJIEIB3ZRUO2JF64ME4KHI6MI43Z4BGPW2LBBQ3WGWTA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.787459380 (567h9m33.140244498s) [upload=499148]: Next returned nil segment Sep 22 05:17:57.787474078 (567h9m33.140259205s) [upload=499148]: Waiting for error group managing segments... Sep 22 05:17:57.797424119 (567h9m33.150209227s) [noise=true , address=77.90.31.12:4592 , piece_id=FOTIXM3HUVN5RSNCONJMDYW2A2FQKJGIFXFH3HPRQMCYDIZ3D2JA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.797935569 (567h9m33.150720677s) [noise=true , address=45.91.116.8:4047 , piece_id=JZNCWA7323AGW3MBN2AFPYOFTUTZHS3ULF23ZKZVIEHVDCVUXMXA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798273803 (567h9m33.151058901s) [noise=true , address=45.91.116.12:4227 , piece_id=W32WUPFKSPWRWBCL7JADKWIU7EQODOI27T7KTPN7WCVCZCUQ4OXQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798418767 (567h9m33.151203875s) [noise=true , address=45.91.116.13:4155 , piece_id=WQYODY6JA56UQYQHVIU4N4OAEVDVUSNM7WEAS3KLA22JXZ3NS73A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798591931 (567h9m33.15137704s) [noise=true , address=45.91.116.5:4299 , piece_id=AIADMFRASUJDIGECRYLT3J3VSCY73HLWLKQGSWIQ27KMNH7ZYKZA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798814057 (567h9m33.151599156s) [noise=true , address=77.90.31.13:4628 , piece_id=ILDBZSYPQSFOJMQJIJ3YY6ZAEGGALRP6NCBFB77BUUZF7EHWDLOQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798903976 (567h9m33.151689084s) [noise=true , address=45.91.116.14:4189 , piece_id=4ZNEJODQNSUDF3AJR3GFEAABDOYPYVJBW4TYH7IXSHOAMLYEU25Q, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799239796 (567h9m33.152024904s) [noise=true , address=45.91.116.4:4937 , piece_id=S4AQXWAPQ6CR42ATHGKIEN36SVHBWNMLBGULGMORWHRCPC2CA4AA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799291223 (567h9m33.152076321s) [noise=true , address=45.91.116.12:4225 , piece_id=NYCEKKNFDFVIYUEJEYJ6TH7ZQWZILNECJUTYZQYTH3H75IG2VRJA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799717784 (567h9m33.152502882s) [noise=true , address=45.91.116.14:4190 , piece_id=QZUCXRPJXFHAQD372QMII7WLQ3EXRMB5TAXHLOFPA6JJJCJSEQ2A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.800251245 (567h9m33.153036353s) [noise=true , address=77.90.31.17:4770 , piece_id=7JWSWRVKROEAHCFZD2CVFRYFIMBYQRS2IH72NQYADFPDVDZ5LDOQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.800580202 (567h9m33.15336531s) [noise=true , address=77.90.31.21:4879 , piece_id=Z7GY3H6OT4BHOABHLY7RXNEEE2LRWQCY2EVEPONLFW6K2YHTBCSA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:18:27.632317337 (567h10m2.985102445s) [noise=true , address=145.79.227.17:4972 , piece_id=YBWZPNXLC7DDYVSQ5UY3JI7FYI3HKRVRRAECT7B4SZ3DQN32GSQA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632381407 (567h10m2.985166515s) [noise=true , address=145.79.227.13:4803 , piece_id=RFE4LY72VJIEIB3ZRUO2JF64ME4KHI6MI43Z4BGPW2LBBQ3WGWTA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632413378 (567h10m2.985198476s) [noise=true , address=145.79.227.16:4909 , piece_id=BKRUZJZ7CVO6XTOS7ASPCRHC3ZHNTZRW62FZDZJ2VKUUFHIAYA4Q, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:27.632470525 (567h10m2.985255623s) [noise=true , address=145.79.227.15:4874 , piece_id=IKJBG6PIHIA6RGACRBBZMEHG5TVWYORKJOUKOPU2C7PXTV2XOJ7A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632487837 (567h10m2.985272945s) [seg_pos={0 0}, upload=499148]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:18:27.648303797 (567h10m3.001088895s) [upload=499148]: Done uploading segments... Sep 22 05:18:27.648323044 (567h10m3.001108141s) [upload=499148]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 527.696574ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X.X..X...............] duration: 400.620557ms FAIL (2025-09-22 05:20:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443356010 (566h58m39.796141139s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443765328 (566h58m39.796550456s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444090558 (566h58m39.796875686s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444188903 (566h58m39.79697403s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444577773 (566h58m39.797362901s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444677179 (566h58m39.797462307s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444962344 (566h58m39.797747472s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445216612 (566h58m39.79800174s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445329904 (566h58m39.798115033s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445688447 (566h58m39.798473585s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445754561 (566h58m39.798539689s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446085091 (566h58m39.79887022s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446552749 (566h58m39.799337917s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447022330 (566h58m39.799807459s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447446877 (566h58m39.800232005s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447878927 (566h58m39.800664056s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.448061300 (566h58m39.800846428s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.528619474 (566h58m39.881404572s) [upload=498993]: Next returned nil segment Sep 22 05:07:04.528630534 (566h58m39.881415643s) [upload=498993]: Waiting for error group managing segments... Sep 22 05:07:04.538575067 (566h58m39.891360164s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539546860 (566h58m39.892331958s) [noise=true , address=145.79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539973170 (566h58m39.892758267s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540146776 (566h58m39.892931913s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540291638 (566h58m39.893076746s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540795884 (566h58m39.893581022s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.544155669 (566h58m39.896940767s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552098143 (566h58m39.904883281s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552115576 (566h58m39.904900674s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.041097924 (566h58m40.393883022s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.506063937 (566h58m40.858849035s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1ejnWsitdrh68vSJvjR2QnRg4kva8DFRzqvvUuDkXM6zKUDWNC, address:145.79.227.11:4736): protocol: rpc: dial tcp 145.79.227.11:4736: connect: no route to host; manager closed: conn closed: op Sep 22 05:08:04.401016602 (566h59m39.75380173s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1YnhzSepDNJ6FEFq6ifLdmaGm7BZhMYs7JWbGmShbi38gSxxXL, address:145.79.227.10:4419): protocol: rpc: dial tcp 145.79.227.10:4419: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401091332 (566h59m39.753876461s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ, address:145.79.227.12:4782): protocol: rpc: dial tcp 145.79.227.12:4782: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401101101 (566h59m39.753886229s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1gakF2Cctzc6EhFScfLfBHJdcDHqMBcey4mNoToEwbtk8Pwix5, address:145.79.227.15:4890): protocol: rpc: dial tcp 145.79.227.15:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401148540 (566h59m39.753933678s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:145.79.227.14:4866): protocol: rpc: dial tcp 145.79.227.14:4866: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401183265 (566h59m39.753968403s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401203854 (566h59m39.753988992s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:145.79.227.2:4118): protocol: rpc: dial tcp 145.79.227.2:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401243117 (566h59m39.754028245s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12mRuEpX7CMxWXVdHrTi3zF3v9t3LcqE6mJKzocajEvYHM52a7T, address:145.79.227.8:4355): protocol: rpc: dial tcp 145.79.227.8:4355: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401288963 (566h59m39.754074102s) [seg_pos={0 0}, upload=498993]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Sep 22 05:08:04.415054225 (566h59m39.767839353s) [upload=498993]: Done uploading segments... Sep 22 05:08:04.415080705 (566h59m39.767865833s) [upload=498993]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 400.646455ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [........................X...X.......X.X.......XXX.X.....X...] duration: 523.269619ms FAIL (2025-09-22 05:20:39 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:17:57.648033125 (567h9m33.000818253s) [upload=499148]: Starting upload Sep 22 05:17:57.648114798 (567h9m33.000899926s) [upload=499148]: Done starting upload Sep 22 05:17:57.648176143 (567h9m33.000961261s) [upload=499148]: Uploading segments... Sep 22 05:17:57.648452281 (567h9m33.001237409s) [upload=499148]: Got next segment. Inline: false Sep 22 05:17:57.648494821 (567h9m33.001279949s) [upload=499148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:17:57.648499850 (567h9m33.001284979s) [upload=499148]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:17:57.670664705 (567h9m33.023449833s) [seg_pos={0 0}, upload=499148]: Begin upload segment... Sep 22 05:17:57.670704479 (567h9m33.023489608s) [seg_pos={0 0}, upload=499148]: Done begin upload segment. Sep 22 05:17:57.670739555 (567h9m33.023524693s) [noise=true , address=45.91.116.12:4225 , piece_id=NYCEKKNFDFVIYUEJEYJ6TH7ZQWZILNECJUTYZQYTH3H75IG2VRJA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671214056 (567h9m33.023999184s) [noise=true , address=77.90.31.13:4628 , piece_id=ILDBZSYPQSFOJMQJIJ3YY6ZAEGGALRP6NCBFB77BUUZF7EHWDLOQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671585262 (567h9m33.0243704s) [noise=true , address=45.91.116.5:4299 , piece_id=AIADMFRASUJDIGECRYLT3J3VSCY73HLWLKQGSWIQ27KMNH7ZYKZA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671997806 (567h9m33.024782934s) [noise=true , address=77.90.31.21:4879 , piece_id=Z7GY3H6OT4BHOABHLY7RXNEEE2LRWQCY2EVEPONLFW6K2YHTBCSA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.672507884 (567h9m33.025293172s) [noise=true , address=77.90.31.12:4592 , piece_id=FOTIXM3HUVN5RSNCONJMDYW2A2FQKJGIFXFH3HPRQMCYDIZ3D2JA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673048238 (567h9m33.025833366s) [noise=true , address=45.91.116.12:4227 , piece_id=W32WUPFKSPWRWBCL7JADKWIU7EQODOI27T7KTPN7WCVCZCUQ4OXQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673455823 (567h9m33.02624096s) [noise=true , address=45.91.116.4:4937 , piece_id=S4AQXWAPQ6CR42ATHGKIEN36SVHBWNMLBGULGMORWHRCPC2CA4AA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673895858 (567h9m33.026680996s) [noise=true , address=45.91.116.13:4155 , piece_id=WQYODY6JA56UQYQHVIU4N4OAEVDVUSNM7WEAS3KLA22JXZ3NS73A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.674287894 (567h9m33.027073022s) [noise=true , address=45.91.116.14:4190 , piece_id=QZUCXRPJXFHAQD372QMII7WLQ3EXRMB5TAXHLOFPA6JJJCJSEQ2A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.674646156 (567h9m33.027431284s) [noise=true , address=45.91.116.8:4047 , piece_id=JZNCWA7323AGW3MBN2AFPYOFTUTZHS3ULF23ZKZVIEHVDCVUXMXA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675012384 (567h9m33.027797512s) [noise=true , address=145.79.227.16:4909 , piece_id=BKRUZJZ7CVO6XTOS7ASPCRHC3ZHNTZRW62FZDZJ2VKUUFHIAYA4Q, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675382799 (567h9m33.028167936s) [noise=true , address=45.91.116.14:4189 , piece_id=4ZNEJODQNSUDF3AJR3GFEAABDOYPYVJBW4TYH7IXSHOAMLYEU25Q, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675735730 (567h9m33.028520859s) [noise=true , address=145.79.227.17:4972 , piece_id=YBWZPNXLC7DDYVSQ5UY3JI7FYI3HKRVRRAECT7B4SZ3DQN32GSQA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676142213 (567h9m33.028927352s) [noise=true , address=145.79.227.15:4874 , piece_id=IKJBG6PIHIA6RGACRBBZMEHG5TVWYORKJOUKOPU2C7PXTV2XOJ7A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676508621 (567h9m33.029293749s) [noise=true , address=77.90.31.17:4770 , piece_id=7JWSWRVKROEAHCFZD2CVFRYFIMBYQRS2IH72NQYADFPDVDZ5LDOQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676883124 (567h9m33.029668252s) [noise=true , address=145.79.227.13:4803 , piece_id=RFE4LY72VJIEIB3ZRUO2JF64ME4KHI6MI43Z4BGPW2LBBQ3WGWTA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.787459380 (567h9m33.140244498s) [upload=499148]: Next returned nil segment Sep 22 05:17:57.787474078 (567h9m33.140259205s) [upload=499148]: Waiting for error group managing segments... Sep 22 05:17:57.797424119 (567h9m33.150209227s) [noise=true , address=77.90.31.12:4592 , piece_id=FOTIXM3HUVN5RSNCONJMDYW2A2FQKJGIFXFH3HPRQMCYDIZ3D2JA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.797935569 (567h9m33.150720677s) [noise=true , address=45.91.116.8:4047 , piece_id=JZNCWA7323AGW3MBN2AFPYOFTUTZHS3ULF23ZKZVIEHVDCVUXMXA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798273803 (567h9m33.151058901s) [noise=true , address=45.91.116.12:4227 , piece_id=W32WUPFKSPWRWBCL7JADKWIU7EQODOI27T7KTPN7WCVCZCUQ4OXQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798418767 (567h9m33.151203875s) [noise=true , address=45.91.116.13:4155 , piece_id=WQYODY6JA56UQYQHVIU4N4OAEVDVUSNM7WEAS3KLA22JXZ3NS73A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798591931 (567h9m33.15137704s) [noise=true , address=45.91.116.5:4299 , piece_id=AIADMFRASUJDIGECRYLT3J3VSCY73HLWLKQGSWIQ27KMNH7ZYKZA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798814057 (567h9m33.151599156s) [noise=true , address=77.90.31.13:4628 , piece_id=ILDBZSYPQSFOJMQJIJ3YY6ZAEGGALRP6NCBFB77BUUZF7EHWDLOQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798903976 (567h9m33.151689084s) [noise=true , address=45.91.116.14:4189 , piece_id=4ZNEJODQNSUDF3AJR3GFEAABDOYPYVJBW4TYH7IXSHOAMLYEU25Q, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799239796 (567h9m33.152024904s) [noise=true , address=45.91.116.4:4937 , piece_id=S4AQXWAPQ6CR42ATHGKIEN36SVHBWNMLBGULGMORWHRCPC2CA4AA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799291223 (567h9m33.152076321s) [noise=true , address=45.91.116.12:4225 , piece_id=NYCEKKNFDFVIYUEJEYJ6TH7ZQWZILNECJUTYZQYTH3H75IG2VRJA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799717784 (567h9m33.152502882s) [noise=true , address=45.91.116.14:4190 , piece_id=QZUCXRPJXFHAQD372QMII7WLQ3EXRMB5TAXHLOFPA6JJJCJSEQ2A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.800251245 (567h9m33.153036353s) [noise=true , address=77.90.31.17:4770 , piece_id=7JWSWRVKROEAHCFZD2CVFRYFIMBYQRS2IH72NQYADFPDVDZ5LDOQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.800580202 (567h9m33.15336531s) [noise=true , address=77.90.31.21:4879 , piece_id=Z7GY3H6OT4BHOABHLY7RXNEEE2LRWQCY2EVEPONLFW6K2YHTBCSA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:18:27.632317337 (567h10m2.985102445s) [noise=true , address=145.79.227.17:4972 , piece_id=YBWZPNXLC7DDYVSQ5UY3JI7FYI3HKRVRRAECT7B4SZ3DQN32GSQA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632381407 (567h10m2.985166515s) [noise=true , address=145.79.227.13:4803 , piece_id=RFE4LY72VJIEIB3ZRUO2JF64ME4KHI6MI43Z4BGPW2LBBQ3WGWTA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632413378 (567h10m2.985198476s) [noise=true , address=145.79.227.16:4909 , piece_id=BKRUZJZ7CVO6XTOS7ASPCRHC3ZHNTZRW62FZDZJ2VKUUFHIAYA4Q, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:27.632470525 (567h10m2.985255623s) [noise=true , address=145.79.227.15:4874 , piece_id=IKJBG6PIHIA6RGACRBBZMEHG5TVWYORKJOUKOPU2C7PXTV2XOJ7A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632487837 (567h10m2.985272945s) [seg_pos={0 0}, upload=499148]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:18:27.648303797 (567h10m3.001088895s) [upload=499148]: Done uploading segments... Sep 22 05:18:27.648323044 (567h10m3.001108141s) [upload=499148]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 523.298051ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXXX.........X........X..................X...X.....X....XXX.] duration: 404.33672ms FAIL (2025-09-22 05:20:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 404.349084ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXXXX.........X........X..................X...X.....X....XXX] duration: 30.000929759s FAIL (2025-09-22 05:20:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000948564s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.121704309s FAIL (2025-09-22 05:20:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.121727863s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X........................X...X.......X.X.......XXX.X.....X..] duration: 404.301004ms FAIL (2025-09-22 05:20:06 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:17:57.648033125 (567h9m33.000818253s) [upload=499148]: Starting upload Sep 22 05:17:57.648114798 (567h9m33.000899926s) [upload=499148]: Done starting upload Sep 22 05:17:57.648176143 (567h9m33.000961261s) [upload=499148]: Uploading segments... Sep 22 05:17:57.648452281 (567h9m33.001237409s) [upload=499148]: Got next segment. Inline: false Sep 22 05:17:57.648494821 (567h9m33.001279949s) [upload=499148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:17:57.648499850 (567h9m33.001284979s) [upload=499148]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:17:57.670664705 (567h9m33.023449833s) [seg_pos={0 0}, upload=499148]: Begin upload segment... Sep 22 05:17:57.670704479 (567h9m33.023489608s) [seg_pos={0 0}, upload=499148]: Done begin upload segment. Sep 22 05:17:57.670739555 (567h9m33.023524693s) [noise=true , address=45.91.116.12:4225 , piece_id=NYCEKKNFDFVIYUEJEYJ6TH7ZQWZILNECJUTYZQYTH3H75IG2VRJA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671214056 (567h9m33.023999184s) [noise=true , address=77.90.31.13:4628 , piece_id=ILDBZSYPQSFOJMQJIJ3YY6ZAEGGALRP6NCBFB77BUUZF7EHWDLOQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671585262 (567h9m33.0243704s) [noise=true , address=45.91.116.5:4299 , piece_id=AIADMFRASUJDIGECRYLT3J3VSCY73HLWLKQGSWIQ27KMNH7ZYKZA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671997806 (567h9m33.024782934s) [noise=true , address=77.90.31.21:4879 , piece_id=Z7GY3H6OT4BHOABHLY7RXNEEE2LRWQCY2EVEPONLFW6K2YHTBCSA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.672507884 (567h9m33.025293172s) [noise=true , address=77.90.31.12:4592 , piece_id=FOTIXM3HUVN5RSNCONJMDYW2A2FQKJGIFXFH3HPRQMCYDIZ3D2JA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673048238 (567h9m33.025833366s) [noise=true , address=45.91.116.12:4227 , piece_id=W32WUPFKSPWRWBCL7JADKWIU7EQODOI27T7KTPN7WCVCZCUQ4OXQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673455823 (567h9m33.02624096s) [noise=true , address=45.91.116.4:4937 , piece_id=S4AQXWAPQ6CR42ATHGKIEN36SVHBWNMLBGULGMORWHRCPC2CA4AA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673895858 (567h9m33.026680996s) [noise=true , address=45.91.116.13:4155 , piece_id=WQYODY6JA56UQYQHVIU4N4OAEVDVUSNM7WEAS3KLA22JXZ3NS73A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.674287894 (567h9m33.027073022s) [noise=true , address=45.91.116.14:4190 , piece_id=QZUCXRPJXFHAQD372QMII7WLQ3EXRMB5TAXHLOFPA6JJJCJSEQ2A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.674646156 (567h9m33.027431284s) [noise=true , address=45.91.116.8:4047 , piece_id=JZNCWA7323AGW3MBN2AFPYOFTUTZHS3ULF23ZKZVIEHVDCVUXMXA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675012384 (567h9m33.027797512s) [noise=true , address=145.79.227.16:4909 , piece_id=BKRUZJZ7CVO6XTOS7ASPCRHC3ZHNTZRW62FZDZJ2VKUUFHIAYA4Q, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675382799 (567h9m33.028167936s) [noise=true , address=45.91.116.14:4189 , piece_id=4ZNEJODQNSUDF3AJR3GFEAABDOYPYVJBW4TYH7IXSHOAMLYEU25Q, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675735730 (567h9m33.028520859s) [noise=true , address=145.79.227.17:4972 , piece_id=YBWZPNXLC7DDYVSQ5UY3JI7FYI3HKRVRRAECT7B4SZ3DQN32GSQA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676142213 (567h9m33.028927352s) [noise=true , address=145.79.227.15:4874 , piece_id=IKJBG6PIHIA6RGACRBBZMEHG5TVWYORKJOUKOPU2C7PXTV2XOJ7A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676508621 (567h9m33.029293749s) [noise=true , address=77.90.31.17:4770 , piece_id=7JWSWRVKROEAHCFZD2CVFRYFIMBYQRS2IH72NQYADFPDVDZ5LDOQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676883124 (567h9m33.029668252s) [noise=true , address=145.79.227.13:4803 , piece_id=RFE4LY72VJIEIB3ZRUO2JF64ME4KHI6MI43Z4BGPW2LBBQ3WGWTA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.787459380 (567h9m33.140244498s) [upload=499148]: Next returned nil segment Sep 22 05:17:57.787474078 (567h9m33.140259205s) [upload=499148]: Waiting for error group managing segments... Sep 22 05:17:57.797424119 (567h9m33.150209227s) [noise=true , address=77.90.31.12:4592 , piece_id=FOTIXM3HUVN5RSNCONJMDYW2A2FQKJGIFXFH3HPRQMCYDIZ3D2JA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.797935569 (567h9m33.150720677s) [noise=true , address=45.91.116.8:4047 , piece_id=JZNCWA7323AGW3MBN2AFPYOFTUTZHS3ULF23ZKZVIEHVDCVUXMXA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798273803 (567h9m33.151058901s) [noise=true , address=45.91.116.12:4227 , piece_id=W32WUPFKSPWRWBCL7JADKWIU7EQODOI27T7KTPN7WCVCZCUQ4OXQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798418767 (567h9m33.151203875s) [noise=true , address=45.91.116.13:4155 , piece_id=WQYODY6JA56UQYQHVIU4N4OAEVDVUSNM7WEAS3KLA22JXZ3NS73A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798591931 (567h9m33.15137704s) [noise=true , address=45.91.116.5:4299 , piece_id=AIADMFRASUJDIGECRYLT3J3VSCY73HLWLKQGSWIQ27KMNH7ZYKZA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798814057 (567h9m33.151599156s) [noise=true , address=77.90.31.13:4628 , piece_id=ILDBZSYPQSFOJMQJIJ3YY6ZAEGGALRP6NCBFB77BUUZF7EHWDLOQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798903976 (567h9m33.151689084s) [noise=true , address=45.91.116.14:4189 , piece_id=4ZNEJODQNSUDF3AJR3GFEAABDOYPYVJBW4TYH7IXSHOAMLYEU25Q, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799239796 (567h9m33.152024904s) [noise=true , address=45.91.116.4:4937 , piece_id=S4AQXWAPQ6CR42ATHGKIEN36SVHBWNMLBGULGMORWHRCPC2CA4AA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799291223 (567h9m33.152076321s) [noise=true , address=45.91.116.12:4225 , piece_id=NYCEKKNFDFVIYUEJEYJ6TH7ZQWZILNECJUTYZQYTH3H75IG2VRJA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799717784 (567h9m33.152502882s) [noise=true , address=45.91.116.14:4190 , piece_id=QZUCXRPJXFHAQD372QMII7WLQ3EXRMB5TAXHLOFPA6JJJCJSEQ2A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.800251245 (567h9m33.153036353s) [noise=true , address=77.90.31.17:4770 , piece_id=7JWSWRVKROEAHCFZD2CVFRYFIMBYQRS2IH72NQYADFPDVDZ5LDOQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.800580202 (567h9m33.15336531s) [noise=true , address=77.90.31.21:4879 , piece_id=Z7GY3H6OT4BHOABHLY7RXNEEE2LRWQCY2EVEPONLFW6K2YHTBCSA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:18:27.632317337 (567h10m2.985102445s) [noise=true , address=145.79.227.17:4972 , piece_id=YBWZPNXLC7DDYVSQ5UY3JI7FYI3HKRVRRAECT7B4SZ3DQN32GSQA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632381407 (567h10m2.985166515s) [noise=true , address=145.79.227.13:4803 , piece_id=RFE4LY72VJIEIB3ZRUO2JF64ME4KHI6MI43Z4BGPW2LBBQ3WGWTA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632413378 (567h10m2.985198476s) [noise=true , address=145.79.227.16:4909 , piece_id=BKRUZJZ7CVO6XTOS7ASPCRHC3ZHNTZRW62FZDZJ2VKUUFHIAYA4Q, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:27.632470525 (567h10m2.985255623s) [noise=true , address=145.79.227.15:4874 , piece_id=IKJBG6PIHIA6RGACRBBZMEHG5TVWYORKJOUKOPU2C7PXTV2XOJ7A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632487837 (567h10m2.985272945s) [seg_pos={0 0}, upload=499148]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:18:27.648303797 (567h10m3.001088895s) [upload=499148]: Done uploading segments... Sep 22 05:18:27.648323044 (567h10m3.001108141s) [upload=499148]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 404.327324ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X.X..X..............] duration: 429.947563ms FAIL (2025-09-22 05:19:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443356010 (566h58m39.796141139s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443765328 (566h58m39.796550456s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444090558 (566h58m39.796875686s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444188903 (566h58m39.79697403s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444577773 (566h58m39.797362901s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444677179 (566h58m39.797462307s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444962344 (566h58m39.797747472s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445216612 (566h58m39.79800174s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445329904 (566h58m39.798115033s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445688447 (566h58m39.798473585s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445754561 (566h58m39.798539689s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446085091 (566h58m39.79887022s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446552749 (566h58m39.799337917s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447022330 (566h58m39.799807459s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447446877 (566h58m39.800232005s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447878927 (566h58m39.800664056s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.448061300 (566h58m39.800846428s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.528619474 (566h58m39.881404572s) [upload=498993]: Next returned nil segment Sep 22 05:07:04.528630534 (566h58m39.881415643s) [upload=498993]: Waiting for error group managing segments... Sep 22 05:07:04.538575067 (566h58m39.891360164s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539546860 (566h58m39.892331958s) [noise=true , address=145.79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539973170 (566h58m39.892758267s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540146776 (566h58m39.892931913s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540291638 (566h58m39.893076746s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540795884 (566h58m39.893581022s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.544155669 (566h58m39.896940767s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552098143 (566h58m39.904883281s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552115576 (566h58m39.904900674s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.041097924 (566h58m40.393883022s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.506063937 (566h58m40.858849035s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1ejnWsitdrh68vSJvjR2QnRg4kva8DFRzqvvUuDkXM6zKUDWNC, address:145.79.227.11:4736): protocol: rpc: dial tcp 145.79.227.11:4736: connect: no route to host; manager closed: conn closed: op Sep 22 05:08:04.401016602 (566h59m39.75380173s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1YnhzSepDNJ6FEFq6ifLdmaGm7BZhMYs7JWbGmShbi38gSxxXL, address:145.79.227.10:4419): protocol: rpc: dial tcp 145.79.227.10:4419: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401091332 (566h59m39.753876461s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ, address:145.79.227.12:4782): protocol: rpc: dial tcp 145.79.227.12:4782: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401101101 (566h59m39.753886229s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1gakF2Cctzc6EhFScfLfBHJdcDHqMBcey4mNoToEwbtk8Pwix5, address:145.79.227.15:4890): protocol: rpc: dial tcp 145.79.227.15:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401148540 (566h59m39.753933678s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:145.79.227.14:4866): protocol: rpc: dial tcp 145.79.227.14:4866: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401183265 (566h59m39.753968403s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401203854 (566h59m39.753988992s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:145.79.227.2:4118): protocol: rpc: dial tcp 145.79.227.2:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401243117 (566h59m39.754028245s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12mRuEpX7CMxWXVdHrTi3zF3v9t3LcqE6mJKzocajEvYHM52a7T, address:145.79.227.8:4355): protocol: rpc: dial tcp 145.79.227.8:4355: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401288963 (566h59m39.754074102s) [seg_pos={0 0}, upload=498993]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Sep 22 05:08:04.415054225 (566h59m39.767839353s) [upload=498993]: Done uploading segments... Sep 22 05:08:04.415080705 (566h59m39.767865833s) [upload=498993]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 429.969273ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XXXXX.........X........X..................X...X.....X....XX] duration: 30.00148497s FAIL (2025-09-22 05:19:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001505027s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.130150183s FAIL (2025-09-22 05:19:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.130170891s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X........................X...X.......X.X.......XXX.X.....X.] duration: 440.162271ms FAIL (2025-09-22 05:18:57 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:17:57.648033125 (567h9m33.000818253s) [upload=499148]: Starting upload Sep 22 05:17:57.648114798 (567h9m33.000899926s) [upload=499148]: Done starting upload Sep 22 05:17:57.648176143 (567h9m33.000961261s) [upload=499148]: Uploading segments... Sep 22 05:17:57.648452281 (567h9m33.001237409s) [upload=499148]: Got next segment. Inline: false Sep 22 05:17:57.648494821 (567h9m33.001279949s) [upload=499148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:17:57.648499850 (567h9m33.001284979s) [upload=499148]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:17:57.670664705 (567h9m33.023449833s) [seg_pos={0 0}, upload=499148]: Begin upload segment... Sep 22 05:17:57.670704479 (567h9m33.023489608s) [seg_pos={0 0}, upload=499148]: Done begin upload segment. Sep 22 05:17:57.670739555 (567h9m33.023524693s) [noise=true , address=45.91.116.12:4225 , piece_id=NYCEKKNFDFVIYUEJEYJ6TH7ZQWZILNECJUTYZQYTH3H75IG2VRJA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671214056 (567h9m33.023999184s) [noise=true , address=77.90.31.13:4628 , piece_id=ILDBZSYPQSFOJMQJIJ3YY6ZAEGGALRP6NCBFB77BUUZF7EHWDLOQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671585262 (567h9m33.0243704s) [noise=true , address=45.91.116.5:4299 , piece_id=AIADMFRASUJDIGECRYLT3J3VSCY73HLWLKQGSWIQ27KMNH7ZYKZA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671997806 (567h9m33.024782934s) [noise=true , address=77.90.31.21:4879 , piece_id=Z7GY3H6OT4BHOABHLY7RXNEEE2LRWQCY2EVEPONLFW6K2YHTBCSA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.672507884 (567h9m33.025293172s) [noise=true , address=77.90.31.12:4592 , piece_id=FOTIXM3HUVN5RSNCONJMDYW2A2FQKJGIFXFH3HPRQMCYDIZ3D2JA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673048238 (567h9m33.025833366s) [noise=true , address=45.91.116.12:4227 , piece_id=W32WUPFKSPWRWBCL7JADKWIU7EQODOI27T7KTPN7WCVCZCUQ4OXQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673455823 (567h9m33.02624096s) [noise=true , address=45.91.116.4:4937 , piece_id=S4AQXWAPQ6CR42ATHGKIEN36SVHBWNMLBGULGMORWHRCPC2CA4AA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673895858 (567h9m33.026680996s) [noise=true , address=45.91.116.13:4155 , piece_id=WQYODY6JA56UQYQHVIU4N4OAEVDVUSNM7WEAS3KLA22JXZ3NS73A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.674287894 (567h9m33.027073022s) [noise=true , address=45.91.116.14:4190 , piece_id=QZUCXRPJXFHAQD372QMII7WLQ3EXRMB5TAXHLOFPA6JJJCJSEQ2A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.674646156 (567h9m33.027431284s) [noise=true , address=45.91.116.8:4047 , piece_id=JZNCWA7323AGW3MBN2AFPYOFTUTZHS3ULF23ZKZVIEHVDCVUXMXA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675012384 (567h9m33.027797512s) [noise=true , address=145.79.227.16:4909 , piece_id=BKRUZJZ7CVO6XTOS7ASPCRHC3ZHNTZRW62FZDZJ2VKUUFHIAYA4Q, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675382799 (567h9m33.028167936s) [noise=true , address=45.91.116.14:4189 , piece_id=4ZNEJODQNSUDF3AJR3GFEAABDOYPYVJBW4TYH7IXSHOAMLYEU25Q, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675735730 (567h9m33.028520859s) [noise=true , address=145.79.227.17:4972 , piece_id=YBWZPNXLC7DDYVSQ5UY3JI7FYI3HKRVRRAECT7B4SZ3DQN32GSQA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676142213 (567h9m33.028927352s) [noise=true , address=145.79.227.15:4874 , piece_id=IKJBG6PIHIA6RGACRBBZMEHG5TVWYORKJOUKOPU2C7PXTV2XOJ7A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676508621 (567h9m33.029293749s) [noise=true , address=77.90.31.17:4770 , piece_id=7JWSWRVKROEAHCFZD2CVFRYFIMBYQRS2IH72NQYADFPDVDZ5LDOQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676883124 (567h9m33.029668252s) [noise=true , address=145.79.227.13:4803 , piece_id=RFE4LY72VJIEIB3ZRUO2JF64ME4KHI6MI43Z4BGPW2LBBQ3WGWTA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.787459380 (567h9m33.140244498s) [upload=499148]: Next returned nil segment Sep 22 05:17:57.787474078 (567h9m33.140259205s) [upload=499148]: Waiting for error group managing segments... Sep 22 05:17:57.797424119 (567h9m33.150209227s) [noise=true , address=77.90.31.12:4592 , piece_id=FOTIXM3HUVN5RSNCONJMDYW2A2FQKJGIFXFH3HPRQMCYDIZ3D2JA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.797935569 (567h9m33.150720677s) [noise=true , address=45.91.116.8:4047 , piece_id=JZNCWA7323AGW3MBN2AFPYOFTUTZHS3ULF23ZKZVIEHVDCVUXMXA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798273803 (567h9m33.151058901s) [noise=true , address=45.91.116.12:4227 , piece_id=W32WUPFKSPWRWBCL7JADKWIU7EQODOI27T7KTPN7WCVCZCUQ4OXQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798418767 (567h9m33.151203875s) [noise=true , address=45.91.116.13:4155 , piece_id=WQYODY6JA56UQYQHVIU4N4OAEVDVUSNM7WEAS3KLA22JXZ3NS73A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798591931 (567h9m33.15137704s) [noise=true , address=45.91.116.5:4299 , piece_id=AIADMFRASUJDIGECRYLT3J3VSCY73HLWLKQGSWIQ27KMNH7ZYKZA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798814057 (567h9m33.151599156s) [noise=true , address=77.90.31.13:4628 , piece_id=ILDBZSYPQSFOJMQJIJ3YY6ZAEGGALRP6NCBFB77BUUZF7EHWDLOQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798903976 (567h9m33.151689084s) [noise=true , address=45.91.116.14:4189 , piece_id=4ZNEJODQNSUDF3AJR3GFEAABDOYPYVJBW4TYH7IXSHOAMLYEU25Q, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799239796 (567h9m33.152024904s) [noise=true , address=45.91.116.4:4937 , piece_id=S4AQXWAPQ6CR42ATHGKIEN36SVHBWNMLBGULGMORWHRCPC2CA4AA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799291223 (567h9m33.152076321s) [noise=true , address=45.91.116.12:4225 , piece_id=NYCEKKNFDFVIYUEJEYJ6TH7ZQWZILNECJUTYZQYTH3H75IG2VRJA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799717784 (567h9m33.152502882s) [noise=true , address=45.91.116.14:4190 , piece_id=QZUCXRPJXFHAQD372QMII7WLQ3EXRMB5TAXHLOFPA6JJJCJSEQ2A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.800251245 (567h9m33.153036353s) [noise=true , address=77.90.31.17:4770 , piece_id=7JWSWRVKROEAHCFZD2CVFRYFIMBYQRS2IH72NQYADFPDVDZ5LDOQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.800580202 (567h9m33.15336531s) [noise=true , address=77.90.31.21:4879 , piece_id=Z7GY3H6OT4BHOABHLY7RXNEEE2LRWQCY2EVEPONLFW6K2YHTBCSA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:18:27.632317337 (567h10m2.985102445s) [noise=true , address=145.79.227.17:4972 , piece_id=YBWZPNXLC7DDYVSQ5UY3JI7FYI3HKRVRRAECT7B4SZ3DQN32GSQA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632381407 (567h10m2.985166515s) [noise=true , address=145.79.227.13:4803 , piece_id=RFE4LY72VJIEIB3ZRUO2JF64ME4KHI6MI43Z4BGPW2LBBQ3WGWTA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632413378 (567h10m2.985198476s) [noise=true , address=145.79.227.16:4909 , piece_id=BKRUZJZ7CVO6XTOS7ASPCRHC3ZHNTZRW62FZDZJ2VKUUFHIAYA4Q, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:27.632470525 (567h10m2.985255623s) [noise=true , address=145.79.227.15:4874 , piece_id=IKJBG6PIHIA6RGACRBBZMEHG5TVWYORKJOUKOPU2C7PXTV2XOJ7A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632487837 (567h10m2.985272945s) [seg_pos={0 0}, upload=499148]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:18:27.648303797 (567h10m3.001088895s) [upload=499148]: Done uploading segments... Sep 22 05:18:27.648323044 (567h10m3.001108141s) [upload=499148]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 440.186807ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 7.048330239s FAIL (2025-09-22 05:18:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.048358132s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.XXXXX.........X........X..................X...X.....X....X] duration: 30.000385186s FAIL (2025-09-22 05:18:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000405374s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X.X..X.............] duration: 413.105998ms FAIL (2025-09-22 05:18:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443356010 (566h58m39.796141139s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443765328 (566h58m39.796550456s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444090558 (566h58m39.796875686s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444188903 (566h58m39.79697403s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444577773 (566h58m39.797362901s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444677179 (566h58m39.797462307s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444962344 (566h58m39.797747472s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445216612 (566h58m39.79800174s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445329904 (566h58m39.798115033s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445688447 (566h58m39.798473585s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445754561 (566h58m39.798539689s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446085091 (566h58m39.79887022s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446552749 (566h58m39.799337917s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447022330 (566h58m39.799807459s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447446877 (566h58m39.800232005s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447878927 (566h58m39.800664056s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.448061300 (566h58m39.800846428s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.528619474 (566h58m39.881404572s) [upload=498993]: Next returned nil segment Sep 22 05:07:04.528630534 (566h58m39.881415643s) [upload=498993]: Waiting for error group managing segments... Sep 22 05:07:04.538575067 (566h58m39.891360164s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539546860 (566h58m39.892331958s) [noise=true , address=145.79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539973170 (566h58m39.892758267s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540146776 (566h58m39.892931913s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540291638 (566h58m39.893076746s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540795884 (566h58m39.893581022s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.544155669 (566h58m39.896940767s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552098143 (566h58m39.904883281s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552115576 (566h58m39.904900674s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.041097924 (566h58m40.393883022s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.506063937 (566h58m40.858849035s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1ejnWsitdrh68vSJvjR2QnRg4kva8DFRzqvvUuDkXM6zKUDWNC, address:145.79.227.11:4736): protocol: rpc: dial tcp 145.79.227.11:4736: connect: no route to host; manager closed: conn closed: op Sep 22 05:08:04.401016602 (566h59m39.75380173s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1YnhzSepDNJ6FEFq6ifLdmaGm7BZhMYs7JWbGmShbi38gSxxXL, address:145.79.227.10:4419): protocol: rpc: dial tcp 145.79.227.10:4419: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401091332 (566h59m39.753876461s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ, address:145.79.227.12:4782): protocol: rpc: dial tcp 145.79.227.12:4782: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401101101 (566h59m39.753886229s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1gakF2Cctzc6EhFScfLfBHJdcDHqMBcey4mNoToEwbtk8Pwix5, address:145.79.227.15:4890): protocol: rpc: dial tcp 145.79.227.15:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401148540 (566h59m39.753933678s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:145.79.227.14:4866): protocol: rpc: dial tcp 145.79.227.14:4866: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401183265 (566h59m39.753968403s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401203854 (566h59m39.753988992s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:145.79.227.2:4118): protocol: rpc: dial tcp 145.79.227.2:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401243117 (566h59m39.754028245s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12mRuEpX7CMxWXVdHrTi3zF3v9t3LcqE6mJKzocajEvYHM52a7T, address:145.79.227.8:4355): protocol: rpc: dial tcp 145.79.227.8:4355: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401288963 (566h59m39.754074102s) [seg_pos={0 0}, upload=498993]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Sep 22 05:08:04.415054225 (566h59m39.767839353s) [upload=498993]: Done uploading segments... Sep 22 05:08:04.415080705 (566h59m39.767865833s) [upload=498993]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 413.132465ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X........................X...X.......X.X.......XXX.X.....X] duration: 30.017197737s FAIL (2025-09-22 05:18:27 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:17:57.648033125 (567h9m33.000818253s) [upload=499148]: Starting upload Sep 22 05:17:57.648114798 (567h9m33.000899926s) [upload=499148]: Done starting upload Sep 22 05:17:57.648176143 (567h9m33.000961261s) [upload=499148]: Uploading segments... Sep 22 05:17:57.648452281 (567h9m33.001237409s) [upload=499148]: Got next segment. Inline: false Sep 22 05:17:57.648494821 (567h9m33.001279949s) [upload=499148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:17:57.648499850 (567h9m33.001284979s) [upload=499148]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:17:57.670664705 (567h9m33.023449833s) [seg_pos={0 0}, upload=499148]: Begin upload segment... Sep 22 05:17:57.670704479 (567h9m33.023489608s) [seg_pos={0 0}, upload=499148]: Done begin upload segment. Sep 22 05:17:57.670739555 (567h9m33.023524693s) [noise=true , address=45.91.116.12:4225 , piece_id=NYCEKKNFDFVIYUEJEYJ6TH7ZQWZILNECJUTYZQYTH3H75IG2VRJA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671214056 (567h9m33.023999184s) [noise=true , address=77.90.31.13:4628 , piece_id=ILDBZSYPQSFOJMQJIJ3YY6ZAEGGALRP6NCBFB77BUUZF7EHWDLOQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671585262 (567h9m33.0243704s) [noise=true , address=45.91.116.5:4299 , piece_id=AIADMFRASUJDIGECRYLT3J3VSCY73HLWLKQGSWIQ27KMNH7ZYKZA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.671997806 (567h9m33.024782934s) [noise=true , address=77.90.31.21:4879 , piece_id=Z7GY3H6OT4BHOABHLY7RXNEEE2LRWQCY2EVEPONLFW6K2YHTBCSA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.672507884 (567h9m33.025293172s) [noise=true , address=77.90.31.12:4592 , piece_id=FOTIXM3HUVN5RSNCONJMDYW2A2FQKJGIFXFH3HPRQMCYDIZ3D2JA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673048238 (567h9m33.025833366s) [noise=true , address=45.91.116.12:4227 , piece_id=W32WUPFKSPWRWBCL7JADKWIU7EQODOI27T7KTPN7WCVCZCUQ4OXQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673455823 (567h9m33.02624096s) [noise=true , address=45.91.116.4:4937 , piece_id=S4AQXWAPQ6CR42ATHGKIEN36SVHBWNMLBGULGMORWHRCPC2CA4AA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.673895858 (567h9m33.026680996s) [noise=true , address=45.91.116.13:4155 , piece_id=WQYODY6JA56UQYQHVIU4N4OAEVDVUSNM7WEAS3KLA22JXZ3NS73A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.674287894 (567h9m33.027073022s) [noise=true , address=45.91.116.14:4190 , piece_id=QZUCXRPJXFHAQD372QMII7WLQ3EXRMB5TAXHLOFPA6JJJCJSEQ2A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.674646156 (567h9m33.027431284s) [noise=true , address=45.91.116.8:4047 , piece_id=JZNCWA7323AGW3MBN2AFPYOFTUTZHS3ULF23ZKZVIEHVDCVUXMXA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675012384 (567h9m33.027797512s) [noise=true , address=145.79.227.16:4909 , piece_id=BKRUZJZ7CVO6XTOS7ASPCRHC3ZHNTZRW62FZDZJ2VKUUFHIAYA4Q, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675382799 (567h9m33.028167936s) [noise=true , address=45.91.116.14:4189 , piece_id=4ZNEJODQNSUDF3AJR3GFEAABDOYPYVJBW4TYH7IXSHOAMLYEU25Q, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.675735730 (567h9m33.028520859s) [noise=true , address=145.79.227.17:4972 , piece_id=YBWZPNXLC7DDYVSQ5UY3JI7FYI3HKRVRRAECT7B4SZ3DQN32GSQA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676142213 (567h9m33.028927352s) [noise=true , address=145.79.227.15:4874 , piece_id=IKJBG6PIHIA6RGACRBBZMEHG5TVWYORKJOUKOPU2C7PXTV2XOJ7A, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676508621 (567h9m33.029293749s) [noise=true , address=77.90.31.17:4770 , piece_id=7JWSWRVKROEAHCFZD2CVFRYFIMBYQRS2IH72NQYADFPDVDZ5LDOQ, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.676883124 (567h9m33.029668252s) [noise=true , address=145.79.227.13:4803 , piece_id=RFE4LY72VJIEIB3ZRUO2JF64ME4KHI6MI43Z4BGPW2LBBQ3WGWTA, seg_pos={0 0}, upload=499148]: Uploading piece... Sep 22 05:17:57.787459380 (567h9m33.140244498s) [upload=499148]: Next returned nil segment Sep 22 05:17:57.787474078 (567h9m33.140259205s) [upload=499148]: Waiting for error group managing segments... Sep 22 05:17:57.797424119 (567h9m33.150209227s) [noise=true , address=77.90.31.12:4592 , piece_id=FOTIXM3HUVN5RSNCONJMDYW2A2FQKJGIFXFH3HPRQMCYDIZ3D2JA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.797935569 (567h9m33.150720677s) [noise=true , address=45.91.116.8:4047 , piece_id=JZNCWA7323AGW3MBN2AFPYOFTUTZHS3ULF23ZKZVIEHVDCVUXMXA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798273803 (567h9m33.151058901s) [noise=true , address=45.91.116.12:4227 , piece_id=W32WUPFKSPWRWBCL7JADKWIU7EQODOI27T7KTPN7WCVCZCUQ4OXQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798418767 (567h9m33.151203875s) [noise=true , address=45.91.116.13:4155 , piece_id=WQYODY6JA56UQYQHVIU4N4OAEVDVUSNM7WEAS3KLA22JXZ3NS73A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798591931 (567h9m33.15137704s) [noise=true , address=45.91.116.5:4299 , piece_id=AIADMFRASUJDIGECRYLT3J3VSCY73HLWLKQGSWIQ27KMNH7ZYKZA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798814057 (567h9m33.151599156s) [noise=true , address=77.90.31.13:4628 , piece_id=ILDBZSYPQSFOJMQJIJ3YY6ZAEGGALRP6NCBFB77BUUZF7EHWDLOQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.798903976 (567h9m33.151689084s) [noise=true , address=45.91.116.14:4189 , piece_id=4ZNEJODQNSUDF3AJR3GFEAABDOYPYVJBW4TYH7IXSHOAMLYEU25Q, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799239796 (567h9m33.152024904s) [noise=true , address=45.91.116.4:4937 , piece_id=S4AQXWAPQ6CR42ATHGKIEN36SVHBWNMLBGULGMORWHRCPC2CA4AA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799291223 (567h9m33.152076321s) [noise=true , address=45.91.116.12:4225 , piece_id=NYCEKKNFDFVIYUEJEYJ6TH7ZQWZILNECJUTYZQYTH3H75IG2VRJA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.799717784 (567h9m33.152502882s) [noise=true , address=45.91.116.14:4190 , piece_id=QZUCXRPJXFHAQD372QMII7WLQ3EXRMB5TAXHLOFPA6JJJCJSEQ2A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.800251245 (567h9m33.153036353s) [noise=true , address=77.90.31.17:4770 , piece_id=7JWSWRVKROEAHCFZD2CVFRYFIMBYQRS2IH72NQYADFPDVDZ5LDOQ, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:17:57.800580202 (567h9m33.15336531s) [noise=true , address=77.90.31.21:4879 , piece_id=Z7GY3H6OT4BHOABHLY7RXNEEE2LRWQCY2EVEPONLFW6K2YHTBCSA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: <nil> Sep 22 05:18:27.632317337 (567h10m2.985102445s) [noise=true , address=145.79.227.17:4972 , piece_id=YBWZPNXLC7DDYVSQ5UY3JI7FYI3HKRVRRAECT7B4SZ3DQN32GSQA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632381407 (567h10m2.985166515s) [noise=true , address=145.79.227.13:4803 , piece_id=RFE4LY72VJIEIB3ZRUO2JF64ME4KHI6MI43Z4BGPW2LBBQ3WGWTA, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632413378 (567h10m2.985198476s) [noise=true , address=145.79.227.16:4909 , piece_id=BKRUZJZ7CVO6XTOS7ASPCRHC3ZHNTZRW62FZDZJ2VKUUFHIAYA4Q, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:27.632470525 (567h10m2.985255623s) [noise=true , address=145.79.227.15:4874 , piece_id=IKJBG6PIHIA6RGACRBBZMEHG5TVWYORKJOUKOPU2C7PXTV2XOJ7A, seg_pos={0 0}, upload=499148]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:18:27.632487837 (567h10m2.985272945s) [seg_pos={0 0}, upload=499148]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:18:27.648303797 (567h10m3.001088895s) [upload=499148]: Done uploading segments... Sep 22 05:18:27.648323044 (567h10m3.001108141s) [upload=499148]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.017218216s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 7.76308176s FAIL (2025-09-22 05:18:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.763111285s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.792829536s FAIL (2025-09-22 05:17:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.792853721s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX.XXXXX.........X........X..................X...X.....X....] duration: 534.651677ms FAIL (2025-09-22 05:17:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 534.670232ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X........................X...X.......X.X.......XXX.X.....] duration: 423.907137ms FAIL (2025-09-22 05:17:31 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Sep 22 05:13:22.974514149 (567h4m58.327299246s) [upload=499082]: Starting upload Sep 22 05:13:22.974553142 (567h4m58.32733825s) [upload=499082]: Done starting upload Sep 22 05:13:22.974795697 (567h4m58.327580796s) [upload=499082]: Uploading segments... Sep 22 05:13:22.974830853 (567h4m58.327615961s) [upload=499082]: Got next segment. Inline: false Sep 22 05:13:22.974859567 (567h4m58.327644675s) [upload=499082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:13:22.974863354 (567h4m58.327648452s) [upload=499082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:13:22.995531139 (567h4m58.348316247s) [seg_pos={0 0}, upload=499082]: Begin upload segment... Sep 22 05:13:22.995558891 (567h4m58.348343999s) [seg_pos={0 0}, upload=499082]: Done begin upload segment. Sep 22 05:13:22.995576263 (567h4m58.348361361s) [noise=true , address=77.90.31.14:4661 , piece_id=BCCLCYQJSDSBEKWVMOJW6VDFKAYXXFCC3WISWFCMH62YRSXPZL4Q, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.995593626 (567h4m58.348378734s) [noise=true , address=45.91.116.5:4298 , piece_id=HZYWW6ZOBVLZ6GGMRGHFT4SDX3T5XFVB4CNHAAPA3DHZDE2VSTLA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.995856730 (567h4m58.348641838s) [noise=true , address=45.91.116.9:4263 , piece_id=EYZH2GFDEPB4YGJRU3LDQKP2DJOV5ME4CDKUSBNSEMJUBXTL3SCA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996051304 (567h4m58.348836403s) [noise=true , address=145.79.227.5:4222 , piece_id=ZVWHNKTH55VY7QSVV7IKTHMZYD3BZURLN2Y45V7LM2WR2ZVKFVCA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996144419 (567h4m58.348929527s) [noise=true , address=77.90.31.12:4589 , piece_id=UKYJOOHDHBE5G3XDLCHRMBYDDNOCD7DLIWJL5VK7KSDI5MNKOH4A, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996371937 (567h4m58.349157035s) [noise=true , address=145.79.227.7:4293 , piece_id=BMK4JUEYIWUOFN5FVW64U5WP6MUBGH7XKBXBIKVM7JE7SRLUUBJA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996425477 (567h4m58.349210576s) [noise=true , address=45.91.116.10:4081 , piece_id=UOD7HX7P5KTA5DLNZTEDLDLUQ7H6AM7Q2ABJPG6MZOIQNQFAMCPA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996621155 (567h4m58.349406252s) [noise=true , address=145.79.227.18:4007 , piece_id=FSL2KKXHZ7YWIFAZ2EVDGVKY5MZQFLBDYXF57OKGNEHPY5IBCYFA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996791824 (567h4m58.349576922s) [noise=true , address=77.90.31.19:4841 , piece_id=H5WPXGJDIHQ66XZSQUJPPX3EXJOFL2KRSTZ4UVR4P2QLU7LAPCYQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996935534 (567h4m58.349720632s) [noise=true , address=145.79.227.16:4909 , piece_id=KX7ICHBES3KTJDVVMW35OQYLVP2P44GWLXDSFX4MPDGGDWNPB5HQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997112546 (567h4m58.349897644s) [noise=true , address=45.91.116.4:4937 , piece_id=FB6MXGDCQFW7ROO223XG7BCIL4Y6DMOT26RT44YMJEVWMV4X4M6A, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997166287 (567h4m58.349951386s) [noise=true , address=145.79.227.6:4257 , piece_id=6IPNPD32GSFAFCREBBLJZ4H6XOLLBNTH6JIPAZWNYNBTGQS2BE7A, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997440823 (567h4m58.350225921s) [noise=true , address=77.90.31.21:4877 , piece_id=AIESKL35MRMMC26KNVQEKJJ5OMKQCAFEJQPQEQMXRGC64UQSNOCQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997445731 (567h4m58.35023083s) [noise=true , address=77.90.31.15:4699 , piece_id=4KPROWW2WVAAQPNEFPZOGFE5RYKAPQE6SU3Y43BNFGV2G52LV6ZQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997681825 (567h4m58.350466933s) [noise=true , address=145.79.227.14:4837 , piece_id=6L6HHGUQOZHMMSUAR3KHC74GSU6V57HNFF2HHL2ANDR62XVSUWEQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997846784 (567h4m58.350631892s) [noise=true , address=145.79.227.13:4801 , piece_id=MSXGDHBKD76BMLOMN6J6EX2ZU2USYSDSEPG3E4MHRRLXCJAY5FWQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:23.081452328 (567h4m58.434237426s) [upload=499082]: Next returned nil segment Sep 22 05:13:23.081461074 (567h4m58.434246172s) [upload=499082]: Waiting for error group managing segments... Sep 22 05:13:23.090125544 (567h4m58.442910641s) [noise=true , address=45.91.116.9:4263 , piece_id=EYZH2GFDEPB4YGJRU3LDQKP2DJOV5ME4CDKUSBNSEMJUBXTL3SCA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.090835847 (567h4m58.443620945s) [noise=true , address=77.90.31.21:4877 , piece_id=AIESKL35MRMMC26KNVQEKJJ5OMKQCAFEJQPQEQMXRGC64UQSNOCQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.092184747 (567h4m58.444969836s) [noise=true , address=77.90.31.12:4589 , piece_id=UKYJOOHDHBE5G3XDLCHRMBYDDNOCD7DLIWJL5VK7KSDI5MNKOH4A, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.092233589 (567h4m58.445018687s) [noise=true , address=77.90.31.15:4699 , piece_id=4KPROWW2WVAAQPNEFPZOGFE5RYKAPQE6SU3Y43BNFGV2G52LV6ZQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.092562237 (567h4m58.445347334s) [noise=true , address=45.91.116.5:4298 , piece_id=HZYWW6ZOBVLZ6GGMRGHFT4SDX3T5XFVB4CNHAAPA3DHZDE2VSTLA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.092602883 (567h4m58.445387981s) [noise=true , address=45.91.116.10:4081 , piece_id=UOD7HX7P5KTA5DLNZTEDLDLUQ7H6AM7Q2ABJPG6MZOIQNQFAMCPA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.092928704 (567h4m58.445713802s) [noise=true , address=45.91.116.4:4937 , piece_id=FB6MXGDCQFW7ROO223XG7BCIL4Y6DMOT26RT44YMJEVWMV4X4M6A, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.094037113 (567h4m58.446822212s) [noise=true , address=77.90.31.19:4841 , piece_id=H5WPXGJDIHQ66XZSQUJPPX3EXJOFL2KRSTZ4UVR4P2QLU7LAPCYQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.094231209 (567h4m58.447016306s) [noise=true , address=77.90.31.14:4661 , piece_id=BCCLCYQJSDSBEKWVMOJW6VDFKAYXXFCC3WISWFCMH62YRSXPZL4Q, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:24.674433201 (567h5m0.027218298s) [noise=true , address=145.79.227.14:4837 , piece_id=6L6HHGUQOZHMMSUAR3KHC74GSU6V57HNFF2HHL2ANDR62XVSUWEQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: connect: no route to host; manager closed: conn closed: op Sep 22 05:13:25.350750932 (567h5m0.703536029s) [noise=true , address=145.79.227.5:4222 , piece_id=ZVWHNKTH55VY7QSVV7IKTHMZYD3BZURLN2Y45V7LM2WR2ZVKFVCA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:13:25.734546118 (567h5m1.087331215s) [noise=true , address=145.79.227.16:4909 , piece_id=KX7ICHBES3KTJDVVMW35OQYLVP2P44GWLXDSFX4MPDGGDWNPB5HQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; manager closed: conn closed: op Sep 22 05:13:52.962101348 (567h5m28.314886457s) [noise=true , address=145.79.227.7:4293 , piece_id=BMK4JUEYIWUOFN5FVW64U5WP6MUBGH7XKBXBIKVM7JE7SRLUUBJA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:13:52.962169526 (567h5m28.314954624s) [noise=true , address=145.79.227.18:4007 , piece_id=FSL2KKXHZ7YWIFAZ2EVDGVKY5MZQFLBDYXF57OKGNEHPY5IBCYFA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:13:52.962192690 (567h5m28.314977788s) [noise=true , address=145.79.227.13:4801 , piece_id=MSXGDHBKD76BMLOMN6J6EX2ZU2USYSDSEPG3E4MHRRLXCJAY5FWQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:13:52.962209011 (567h5m28.314994119s) [noise=true , address=145.79.227.6:4257 , piece_id=6IPNPD32GSFAFCREBBLJZ4H6XOLLBNTH6JIPAZWNYNBTGQS2BE7A, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:13:52.962229659 (567h5m28.315014767s) [seg_pos={0 0}, upload=499082]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 22 05:13:52.976475153 (567h5m28.329260251s) [upload=499082]: Done uploading segments... Sep 22 05:13:52.976487135 (567h5m28.329272243s) [upload=499082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 423.925221ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X.X..X............] duration: 380.406694ms FAIL (2025-09-22 05:17:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443356010 (566h58m39.796141139s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443765328 (566h58m39.796550456s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444090558 (566h58m39.796875686s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444188903 (566h58m39.79697403s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444577773 (566h58m39.797362901s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444677179 (566h58m39.797462307s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444962344 (566h58m39.797747472s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445216612 (566h58m39.79800174s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445329904 (566h58m39.798115033s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445688447 (566h58m39.798473585s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445754561 (566h58m39.798539689s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446085091 (566h58m39.79887022s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446552749 (566h58m39.799337917s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447022330 (566h58m39.799807459s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447446877 (566h58m39.800232005s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447878927 (566h58m39.800664056s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.448061300 (566h58m39.800846428s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.528619474 (566h58m39.881404572s) [upload=498993]: Next returned nil segment Sep 22 05:07:04.528630534 (566h58m39.881415643s) [upload=498993]: Waiting for error group managing segments... Sep 22 05:07:04.538575067 (566h58m39.891360164s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539546860 (566h58m39.892331958s) [noise=true , address=145.79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539973170 (566h58m39.892758267s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540146776 (566h58m39.892931913s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540291638 (566h58m39.893076746s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540795884 (566h58m39.893581022s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.544155669 (566h58m39.896940767s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552098143 (566h58m39.904883281s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552115576 (566h58m39.904900674s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.041097924 (566h58m40.393883022s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.506063937 (566h58m40.858849035s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1ejnWsitdrh68vSJvjR2QnRg4kva8DFRzqvvUuDkXM6zKUDWNC, address:145.79.227.11:4736): protocol: rpc: dial tcp 145.79.227.11:4736: connect: no route to host; manager closed: conn closed: op Sep 22 05:08:04.401016602 (566h59m39.75380173s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1YnhzSepDNJ6FEFq6ifLdmaGm7BZhMYs7JWbGmShbi38gSxxXL, address:145.79.227.10:4419): protocol: rpc: dial tcp 145.79.227.10:4419: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401091332 (566h59m39.753876461s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ, address:145.79.227.12:4782): protocol: rpc: dial tcp 145.79.227.12:4782: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401101101 (566h59m39.753886229s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1gakF2Cctzc6EhFScfLfBHJdcDHqMBcey4mNoToEwbtk8Pwix5, address:145.79.227.15:4890): protocol: rpc: dial tcp 145.79.227.15:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401148540 (566h59m39.753933678s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:145.79.227.14:4866): protocol: rpc: dial tcp 145.79.227.14:4866: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401183265 (566h59m39.753968403s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401203854 (566h59m39.753988992s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:145.79.227.2:4118): protocol: rpc: dial tcp 145.79.227.2:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401243117 (566h59m39.754028245s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12mRuEpX7CMxWXVdHrTi3zF3v9t3LcqE6mJKzocajEvYHM52a7T, address:145.79.227.8:4355): protocol: rpc: dial tcp 145.79.227.8:4355: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401288963 (566h59m39.754074102s) [seg_pos={0 0}, upload=498993]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Sep 22 05:08:04.415054225 (566h59m39.767839353s) [upload=498993]: Done uploading segments... Sep 22 05:08:04.415080705 (566h59m39.767865833s) [upload=498993]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 380.419608ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXX.XXXXX.........X........X..................X...X.....X...] duration: 648.244614ms FAIL (2025-09-22 05:17:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 648.267448ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.966867894s FAIL (2025-09-22 05:17:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.966893723s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X........................X...X.......X.X.......XXX.X....] duration: 455.949086ms FAIL (2025-09-22 05:16:37 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Sep 22 05:13:22.974514149 (567h4m58.327299246s) [upload=499082]: Starting upload Sep 22 05:13:22.974553142 (567h4m58.32733825s) [upload=499082]: Done starting upload Sep 22 05:13:22.974795697 (567h4m58.327580796s) [upload=499082]: Uploading segments... Sep 22 05:13:22.974830853 (567h4m58.327615961s) [upload=499082]: Got next segment. Inline: false Sep 22 05:13:22.974859567 (567h4m58.327644675s) [upload=499082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:13:22.974863354 (567h4m58.327648452s) [upload=499082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:13:22.995531139 (567h4m58.348316247s) [seg_pos={0 0}, upload=499082]: Begin upload segment... Sep 22 05:13:22.995558891 (567h4m58.348343999s) [seg_pos={0 0}, upload=499082]: Done begin upload segment. Sep 22 05:13:22.995576263 (567h4m58.348361361s) [noise=true , address=77.90.31.14:4661 , piece_id=BCCLCYQJSDSBEKWVMOJW6VDFKAYXXFCC3WISWFCMH62YRSXPZL4Q, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.995593626 (567h4m58.348378734s) [noise=true , address=45.91.116.5:4298 , piece_id=HZYWW6ZOBVLZ6GGMRGHFT4SDX3T5XFVB4CNHAAPA3DHZDE2VSTLA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.995856730 (567h4m58.348641838s) [noise=true , address=45.91.116.9:4263 , piece_id=EYZH2GFDEPB4YGJRU3LDQKP2DJOV5ME4CDKUSBNSEMJUBXTL3SCA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996051304 (567h4m58.348836403s) [noise=true , address=145.79.227.5:4222 , piece_id=ZVWHNKTH55VY7QSVV7IKTHMZYD3BZURLN2Y45V7LM2WR2ZVKFVCA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996144419 (567h4m58.348929527s) [noise=true , address=77.90.31.12:4589 , piece_id=UKYJOOHDHBE5G3XDLCHRMBYDDNOCD7DLIWJL5VK7KSDI5MNKOH4A, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996371937 (567h4m58.349157035s) [noise=true , address=145.79.227.7:4293 , piece_id=BMK4JUEYIWUOFN5FVW64U5WP6MUBGH7XKBXBIKVM7JE7SRLUUBJA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996425477 (567h4m58.349210576s) [noise=true , address=45.91.116.10:4081 , piece_id=UOD7HX7P5KTA5DLNZTEDLDLUQ7H6AM7Q2ABJPG6MZOIQNQFAMCPA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996621155 (567h4m58.349406252s) [noise=true , address=145.79.227.18:4007 , piece_id=FSL2KKXHZ7YWIFAZ2EVDGVKY5MZQFLBDYXF57OKGNEHPY5IBCYFA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996791824 (567h4m58.349576922s) [noise=true , address=77.90.31.19:4841 , piece_id=H5WPXGJDIHQ66XZSQUJPPX3EXJOFL2KRSTZ4UVR4P2QLU7LAPCYQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996935534 (567h4m58.349720632s) [noise=true , address=145.79.227.16:4909 , piece_id=KX7ICHBES3KTJDVVMW35OQYLVP2P44GWLXDSFX4MPDGGDWNPB5HQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997112546 (567h4m58.349897644s) [noise=true , address=45.91.116.4:4937 , piece_id=FB6MXGDCQFW7ROO223XG7BCIL4Y6DMOT26RT44YMJEVWMV4X4M6A, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997166287 (567h4m58.349951386s) [noise=true , address=145.79.227.6:4257 , piece_id=6IPNPD32GSFAFCREBBLJZ4H6XOLLBNTH6JIPAZWNYNBTGQS2BE7A, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997440823 (567h4m58.350225921s) [noise=true , address=77.90.31.21:4877 , piece_id=AIESKL35MRMMC26KNVQEKJJ5OMKQCAFEJQPQEQMXRGC64UQSNOCQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997445731 (567h4m58.35023083s) [noise=true , address=77.90.31.15:4699 , piece_id=4KPROWW2WVAAQPNEFPZOGFE5RYKAPQE6SU3Y43BNFGV2G52LV6ZQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997681825 (567h4m58.350466933s) [noise=true , address=145.79.227.14:4837 , piece_id=6L6HHGUQOZHMMSUAR3KHC74GSU6V57HNFF2HHL2ANDR62XVSUWEQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997846784 (567h4m58.350631892s) [noise=true , address=145.79.227.13:4801 , piece_id=MSXGDHBKD76BMLOMN6J6EX2ZU2USYSDSEPG3E4MHRRLXCJAY5FWQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:23.081452328 (567h4m58.434237426s) [upload=499082]: Next returned nil segment Sep 22 05:13:23.081461074 (567h4m58.434246172s) [upload=499082]: Waiting for error group managing segments... Sep 22 05:13:23.090125544 (567h4m58.442910641s) [noise=true , address=45.91.116.9:4263 , piece_id=EYZH2GFDEPB4YGJRU3LDQKP2DJOV5ME4CDKUSBNSEMJUBXTL3SCA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.090835847 (567h4m58.443620945s) [noise=true , address=77.90.31.21:4877 , piece_id=AIESKL35MRMMC26KNVQEKJJ5OMKQCAFEJQPQEQMXRGC64UQSNOCQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.092184747 (567h4m58.444969836s) [noise=true , address=77.90.31.12:4589 , piece_id=UKYJOOHDHBE5G3XDLCHRMBYDDNOCD7DLIWJL5VK7KSDI5MNKOH4A, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.092233589 (567h4m58.445018687s) [noise=true , address=77.90.31.15:4699 , piece_id=4KPROWW2WVAAQPNEFPZOGFE5RYKAPQE6SU3Y43BNFGV2G52LV6ZQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.092562237 (567h4m58.445347334s) [noise=true , address=45.91.116.5:4298 , piece_id=HZYWW6ZOBVLZ6GGMRGHFT4SDX3T5XFVB4CNHAAPA3DHZDE2VSTLA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.092602883 (567h4m58.445387981s) [noise=true , address=45.91.116.10:4081 , piece_id=UOD7HX7P5KTA5DLNZTEDLDLUQ7H6AM7Q2ABJPG6MZOIQNQFAMCPA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.092928704 (567h4m58.445713802s) [noise=true , address=45.91.116.4:4937 , piece_id=FB6MXGDCQFW7ROO223XG7BCIL4Y6DMOT26RT44YMJEVWMV4X4M6A, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.094037113 (567h4m58.446822212s) [noise=true , address=77.90.31.19:4841 , piece_id=H5WPXGJDIHQ66XZSQUJPPX3EXJOFL2KRSTZ4UVR4P2QLU7LAPCYQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.094231209 (567h4m58.447016306s) [noise=true , address=77.90.31.14:4661 , piece_id=BCCLCYQJSDSBEKWVMOJW6VDFKAYXXFCC3WISWFCMH62YRSXPZL4Q, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:24.674433201 (567h5m0.027218298s) [noise=true , address=145.79.227.14:4837 , piece_id=6L6HHGUQOZHMMSUAR3KHC74GSU6V57HNFF2HHL2ANDR62XVSUWEQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: connect: no route to host; manager closed: conn closed: op Sep 22 05:13:25.350750932 (567h5m0.703536029s) [noise=true , address=145.79.227.5:4222 , piece_id=ZVWHNKTH55VY7QSVV7IKTHMZYD3BZURLN2Y45V7LM2WR2ZVKFVCA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:13:25.734546118 (567h5m1.087331215s) [noise=true , address=145.79.227.16:4909 , piece_id=KX7ICHBES3KTJDVVMW35OQYLVP2P44GWLXDSFX4MPDGGDWNPB5HQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; manager closed: conn closed: op Sep 22 05:13:52.962101348 (567h5m28.314886457s) [noise=true , address=145.79.227.7:4293 , piece_id=BMK4JUEYIWUOFN5FVW64U5WP6MUBGH7XKBXBIKVM7JE7SRLUUBJA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:13:52.962169526 (567h5m28.314954624s) [noise=true , address=145.79.227.18:4007 , piece_id=FSL2KKXHZ7YWIFAZ2EVDGVKY5MZQFLBDYXF57OKGNEHPY5IBCYFA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:13:52.962192690 (567h5m28.314977788s) [noise=true , address=145.79.227.13:4801 , piece_id=MSXGDHBKD76BMLOMN6J6EX2ZU2USYSDSEPG3E4MHRRLXCJAY5FWQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:13:52.962209011 (567h5m28.314994119s) [noise=true , address=145.79.227.6:4257 , piece_id=6IPNPD32GSFAFCREBBLJZ4H6XOLLBNTH6JIPAZWNYNBTGQS2BE7A, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:13:52.962229659 (567h5m28.315014767s) [seg_pos={0 0}, upload=499082]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 22 05:13:52.976475153 (567h5m28.329260251s) [upload=499082]: Done uploading segments... Sep 22 05:13:52.976487135 (567h5m28.329272243s) [upload=499082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 455.981086ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.077028531s FAIL (2025-09-22 05:16:36 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.077056063s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XXX.XXXXX.........X........X..................X...X.....X..] duration: 434.466642ms FAIL (2025-09-22 05:16:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 434.485629ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X.X..X...........] duration: 523.598374ms FAIL (2025-09-22 05:16:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443356010 (566h58m39.796141139s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443765328 (566h58m39.796550456s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444090558 (566h58m39.796875686s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444188903 (566h58m39.79697403s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444577773 (566h58m39.797362901s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444677179 (566h58m39.797462307s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444962344 (566h58m39.797747472s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445216612 (566h58m39.79800174s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445329904 (566h58m39.798115033s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445688447 (566h58m39.798473585s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445754561 (566h58m39.798539689s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446085091 (566h58m39.79887022s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446552749 (566h58m39.799337917s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447022330 (566h58m39.799807459s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447446877 (566h58m39.800232005s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447878927 (566h58m39.800664056s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.448061300 (566h58m39.800846428s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.528619474 (566h58m39.881404572s) [upload=498993]: Next returned nil segment Sep 22 05:07:04.528630534 (566h58m39.881415643s) [upload=498993]: Waiting for error group managing segments... Sep 22 05:07:04.538575067 (566h58m39.891360164s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539546860 (566h58m39.892331958s) [noise=true , address=145.79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539973170 (566h58m39.892758267s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540146776 (566h58m39.892931913s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540291638 (566h58m39.893076746s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540795884 (566h58m39.893581022s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.544155669 (566h58m39.896940767s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552098143 (566h58m39.904883281s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552115576 (566h58m39.904900674s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.041097924 (566h58m40.393883022s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.506063937 (566h58m40.858849035s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1ejnWsitdrh68vSJvjR2QnRg4kva8DFRzqvvUuDkXM6zKUDWNC, address:145.79.227.11:4736): protocol: rpc: dial tcp 145.79.227.11:4736: connect: no route to host; manager closed: conn closed: op Sep 22 05:08:04.401016602 (566h59m39.75380173s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1YnhzSepDNJ6FEFq6ifLdmaGm7BZhMYs7JWbGmShbi38gSxxXL, address:145.79.227.10:4419): protocol: rpc: dial tcp 145.79.227.10:4419: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401091332 (566h59m39.753876461s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ, address:145.79.227.12:4782): protocol: rpc: dial tcp 145.79.227.12:4782: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401101101 (566h59m39.753886229s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1gakF2Cctzc6EhFScfLfBHJdcDHqMBcey4mNoToEwbtk8Pwix5, address:145.79.227.15:4890): protocol: rpc: dial tcp 145.79.227.15:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401148540 (566h59m39.753933678s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:145.79.227.14:4866): protocol: rpc: dial tcp 145.79.227.14:4866: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401183265 (566h59m39.753968403s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401203854 (566h59m39.753988992s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:145.79.227.2:4118): protocol: rpc: dial tcp 145.79.227.2:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401243117 (566h59m39.754028245s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12mRuEpX7CMxWXVdHrTi3zF3v9t3LcqE6mJKzocajEvYHM52a7T, address:145.79.227.8:4355): protocol: rpc: dial tcp 145.79.227.8:4355: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401288963 (566h59m39.754074102s) [seg_pos={0 0}, upload=498993]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Sep 22 05:08:04.415054225 (566h59m39.767839353s) [upload=498993]: Done uploading segments... Sep 22 05:08:04.415080705 (566h59m39.767865833s) [upload=498993]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 523.621617ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.413782217s FAIL (2025-09-22 05:15:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.413809869s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X........................X...X.......X.X.......XXX.X...] duration: 365.666574ms FAIL (2025-09-22 05:15:42 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Sep 22 05:13:22.974514149 (567h4m58.327299246s) [upload=499082]: Starting upload Sep 22 05:13:22.974553142 (567h4m58.32733825s) [upload=499082]: Done starting upload Sep 22 05:13:22.974795697 (567h4m58.327580796s) [upload=499082]: Uploading segments... Sep 22 05:13:22.974830853 (567h4m58.327615961s) [upload=499082]: Got next segment. Inline: false Sep 22 05:13:22.974859567 (567h4m58.327644675s) [upload=499082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:13:22.974863354 (567h4m58.327648452s) [upload=499082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:13:22.995531139 (567h4m58.348316247s) [seg_pos={0 0}, upload=499082]: Begin upload segment... Sep 22 05:13:22.995558891 (567h4m58.348343999s) [seg_pos={0 0}, upload=499082]: Done begin upload segment. Sep 22 05:13:22.995576263 (567h4m58.348361361s) [noise=true , address=77.90.31.14:4661 , piece_id=BCCLCYQJSDSBEKWVMOJW6VDFKAYXXFCC3WISWFCMH62YRSXPZL4Q, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.995593626 (567h4m58.348378734s) [noise=true , address=45.91.116.5:4298 , piece_id=HZYWW6ZOBVLZ6GGMRGHFT4SDX3T5XFVB4CNHAAPA3DHZDE2VSTLA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.995856730 (567h4m58.348641838s) [noise=true , address=45.91.116.9:4263 , piece_id=EYZH2GFDEPB4YGJRU3LDQKP2DJOV5ME4CDKUSBNSEMJUBXTL3SCA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996051304 (567h4m58.348836403s) [noise=true , address=145.79.227.5:4222 , piece_id=ZVWHNKTH55VY7QSVV7IKTHMZYD3BZURLN2Y45V7LM2WR2ZVKFVCA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996144419 (567h4m58.348929527s) [noise=true , address=77.90.31.12:4589 , piece_id=UKYJOOHDHBE5G3XDLCHRMBYDDNOCD7DLIWJL5VK7KSDI5MNKOH4A, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996371937 (567h4m58.349157035s) [noise=true , address=145.79.227.7:4293 , piece_id=BMK4JUEYIWUOFN5FVW64U5WP6MUBGH7XKBXBIKVM7JE7SRLUUBJA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996425477 (567h4m58.349210576s) [noise=true , address=45.91.116.10:4081 , piece_id=UOD7HX7P5KTA5DLNZTEDLDLUQ7H6AM7Q2ABJPG6MZOIQNQFAMCPA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996621155 (567h4m58.349406252s) [noise=true , address=145.79.227.18:4007 , piece_id=FSL2KKXHZ7YWIFAZ2EVDGVKY5MZQFLBDYXF57OKGNEHPY5IBCYFA, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996791824 (567h4m58.349576922s) [noise=true , address=77.90.31.19:4841 , piece_id=H5WPXGJDIHQ66XZSQUJPPX3EXJOFL2KRSTZ4UVR4P2QLU7LAPCYQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.996935534 (567h4m58.349720632s) [noise=true , address=145.79.227.16:4909 , piece_id=KX7ICHBES3KTJDVVMW35OQYLVP2P44GWLXDSFX4MPDGGDWNPB5HQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997112546 (567h4m58.349897644s) [noise=true , address=45.91.116.4:4937 , piece_id=FB6MXGDCQFW7ROO223XG7BCIL4Y6DMOT26RT44YMJEVWMV4X4M6A, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997166287 (567h4m58.349951386s) [noise=true , address=145.79.227.6:4257 , piece_id=6IPNPD32GSFAFCREBBLJZ4H6XOLLBNTH6JIPAZWNYNBTGQS2BE7A, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997440823 (567h4m58.350225921s) [noise=true , address=77.90.31.21:4877 , piece_id=AIESKL35MRMMC26KNVQEKJJ5OMKQCAFEJQPQEQMXRGC64UQSNOCQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997445731 (567h4m58.35023083s) [noise=true , address=77.90.31.15:4699 , piece_id=4KPROWW2WVAAQPNEFPZOGFE5RYKAPQE6SU3Y43BNFGV2G52LV6ZQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997681825 (567h4m58.350466933s) [noise=true , address=145.79.227.14:4837 , piece_id=6L6HHGUQOZHMMSUAR3KHC74GSU6V57HNFF2HHL2ANDR62XVSUWEQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:22.997846784 (567h4m58.350631892s) [noise=true , address=145.79.227.13:4801 , piece_id=MSXGDHBKD76BMLOMN6J6EX2ZU2USYSDSEPG3E4MHRRLXCJAY5FWQ, seg_pos={0 0}, upload=499082]: Uploading piece... Sep 22 05:13:23.081452328 (567h4m58.434237426s) [upload=499082]: Next returned nil segment Sep 22 05:13:23.081461074 (567h4m58.434246172s) [upload=499082]: Waiting for error group managing segments... Sep 22 05:13:23.090125544 (567h4m58.442910641s) [noise=true , address=45.91.116.9:4263 , piece_id=EYZH2GFDEPB4YGJRU3LDQKP2DJOV5ME4CDKUSBNSEMJUBXTL3SCA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.090835847 (567h4m58.443620945s) [noise=true , address=77.90.31.21:4877 , piece_id=AIESKL35MRMMC26KNVQEKJJ5OMKQCAFEJQPQEQMXRGC64UQSNOCQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.092184747 (567h4m58.444969836s) [noise=true , address=77.90.31.12:4589 , piece_id=UKYJOOHDHBE5G3XDLCHRMBYDDNOCD7DLIWJL5VK7KSDI5MNKOH4A, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.092233589 (567h4m58.445018687s) [noise=true , address=77.90.31.15:4699 , piece_id=4KPROWW2WVAAQPNEFPZOGFE5RYKAPQE6SU3Y43BNFGV2G52LV6ZQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.092562237 (567h4m58.445347334s) [noise=true , address=45.91.116.5:4298 , piece_id=HZYWW6ZOBVLZ6GGMRGHFT4SDX3T5XFVB4CNHAAPA3DHZDE2VSTLA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.092602883 (567h4m58.445387981s) [noise=true , address=45.91.116.10:4081 , piece_id=UOD7HX7P5KTA5DLNZTEDLDLUQ7H6AM7Q2ABJPG6MZOIQNQFAMCPA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.092928704 (567h4m58.445713802s) [noise=true , address=45.91.116.4:4937 , piece_id=FB6MXGDCQFW7ROO223XG7BCIL4Y6DMOT26RT44YMJEVWMV4X4M6A, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.094037113 (567h4m58.446822212s) [noise=true , address=77.90.31.19:4841 , piece_id=H5WPXGJDIHQ66XZSQUJPPX3EXJOFL2KRSTZ4UVR4P2QLU7LAPCYQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:23.094231209 (567h4m58.447016306s) [noise=true , address=77.90.31.14:4661 , piece_id=BCCLCYQJSDSBEKWVMOJW6VDFKAYXXFCC3WISWFCMH62YRSXPZL4Q, seg_pos={0 0}, upload=499082]: Done uploading piece. err: <nil> Sep 22 05:13:24.674433201 (567h5m0.027218298s) [noise=true , address=145.79.227.14:4837 , piece_id=6L6HHGUQOZHMMSUAR3KHC74GSU6V57HNFF2HHL2ANDR62XVSUWEQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: connect: no route to host; manager closed: conn closed: op Sep 22 05:13:25.350750932 (567h5m0.703536029s) [noise=true , address=145.79.227.5:4222 , piece_id=ZVWHNKTH55VY7QSVV7IKTHMZYD3BZURLN2Y45V7LM2WR2ZVKFVCA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:13:25.734546118 (567h5m1.087331215s) [noise=true , address=145.79.227.16:4909 , piece_id=KX7ICHBES3KTJDVVMW35OQYLVP2P44GWLXDSFX4MPDGGDWNPB5HQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; manager closed: conn closed: op Sep 22 05:13:52.962101348 (567h5m28.314886457s) [noise=true , address=145.79.227.7:4293 , piece_id=BMK4JUEYIWUOFN5FVW64U5WP6MUBGH7XKBXBIKVM7JE7SRLUUBJA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:13:52.962169526 (567h5m28.314954624s) [noise=true , address=145.79.227.18:4007 , piece_id=FSL2KKXHZ7YWIFAZ2EVDGVKY5MZQFLBDYXF57OKGNEHPY5IBCYFA, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:13:52.962192690 (567h5m28.314977788s) [noise=true , address=145.79.227.13:4801 , piece_id=MSXGDHBKD76BMLOMN6J6EX2ZU2USYSDSEPG3E4MHRRLXCJAY5FWQ, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:13:52.962209011 (567h5m28.314994119s) [noise=true , address=145.79.227.6:4257 , piece_id=6IPNPD32GSFAFCREBBLJZ4H6XOLLBNTH6JIPAZWNYNBTGQS2BE7A, seg_pos={0 0}, upload=499082]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:13:52.962229659 (567h5m28.315014767s) [seg_pos={0 0}, upload=499082]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 22 05:13:52.976475153 (567h5m28.329260251s) [upload=499082]: Done uploading segments... Sep 22 05:13:52.976487135 (567h5m28.329272243s) [upload=499082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 365.693163ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XXX.XXXXX.........X........X..................X...X.....X.] duration: 617.391686ms FAIL (2025-09-22 05:15:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 617.409249ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...XXX.XXXXX.........X........X..................X...X.....X] duration: 30.000481688s FAIL (2025-09-22 05:15:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000504401s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X.X..X..........] duration: 341.180824ms FAIL (2025-09-22 05:15:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443356010 (566h58m39.796141139s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.443765328 (566h58m39.796550456s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444090558 (566h58m39.796875686s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444188903 (566h58m39.79697403s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444577773 (566h58m39.797362901s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444677179 (566h58m39.797462307s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.444962344 (566h58m39.797747472s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445216612 (566h58m39.79800174s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445329904 (566h58m39.798115033s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445688447 (566h58m39.798473585s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.445754561 (566h58m39.798539689s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446085091 (566h58m39.79887022s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.446552749 (566h58m39.799337917s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447022330 (566h58m39.799807459s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447446877 (566h58m39.800232005s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.447878927 (566h58m39.800664056s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.448061300 (566h58m39.800846428s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Uploading piece... Sep 22 05:07:04.528619474 (566h58m39.881404572s) [upload=498993]: Next returned nil segment Sep 22 05:07:04.528630534 (566h58m39.881415643s) [upload=498993]: Waiting for error group managing segments... Sep 22 05:07:04.538575067 (566h58m39.891360164s) [noise=true , address=145.79.227.30:4790 , piece_id=NFLSQZPJ6PTFYTBRAZGGIZWJAJHYPSTIKYCHGMOK3NV562QQDX4A, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539546860 (566h58m39.892331958s) [noise=true , address=145.79.227.27:4717 , piece_id=BJYH4RBBMPRDAB7I3S64JA2RKDSZWEW5WZESL4UBGMHIJ3DJZOFA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.539973170 (566h58m39.892758267s) [noise=true , address=77.90.31.18:4831 , piece_id=6YYF7O7WASB7L4BUMO6FRX3NGH3QPEB2EDMGZ52RUZYPEXI2D5CQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540146776 (566h58m39.892931913s) [noise=true , address=145.79.227.33:4892 , piece_id=IRPSKGA5QTCEYH3H22ZGWUB4PSLWSKDOUGMXH7LAGRTZGO6NEYMQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540291638 (566h58m39.893076746s) [noise=true , address=145.79.227.29:4973 , piece_id=7EWN3F2YPAAAMXRUZHQ7FPYME3PCS7YABMIWY75JXHY2E4PN52FQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.540795884 (566h58m39.893581022s) [noise=true , address=145.79.227.52:4474 , piece_id=AJ55FCPIVDMXVKKZ4VEX5DSJDHENFIWPYEXBRXZUAFD4FH2ERM3Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.544155669 (566h58m39.896940767s) [noise=true , address=145.79.227.25:4629 , piece_id=FE4IVKMTA2UCEVM3IFR6CBPRXIRK7CFJMV7UPO4NULLDPEMHI5GQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552098143 (566h58m39.904883281s) [noise=true , address=145.79.227.46:4624 , piece_id=RLMCJZCLQM62QP3XQLDDQ22CLWHRASVOLU5CDXQVDPQ7ABHESU5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:04.552115576 (566h58m39.904900674s) [noise=true , address=145.79.227.22:4518 , piece_id=QH2EP26EVUZVNQ4U3CJKZD5HLJ4OTLMDPZEVOW7SWA3RGRMZPVJA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.041097924 (566h58m40.393883022s) [noise=true , address=66.23.193.175:4553 , piece_id=CUWCYLLGC2OZTEVGQTZH4H5DQWPRNZSNGW7LMZNM6U5KJI7ZCCIA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: <nil> Sep 22 05:07:05.506063937 (566h58m40.858849035s) [noise=true , address=145.79.227.11:4736 , piece_id=DCW7GQ2BVMIED4W5FNS33XEDUEIT232O5NHZHGFPGGEFRL6IK3YQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1ejnWsitdrh68vSJvjR2QnRg4kva8DFRzqvvUuDkXM6zKUDWNC, address:145.79.227.11:4736): protocol: rpc: dial tcp 145.79.227.11:4736: connect: no route to host; manager closed: conn closed: op Sep 22 05:08:04.401016602 (566h59m39.75380173s) [noise=true , address=145.79.227.10:4419 , piece_id=JB2SSRHP2V2URKBPHPPLPML5SFLEUSF3WBA3HVLX3T5M7DK6TP5Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1YnhzSepDNJ6FEFq6ifLdmaGm7BZhMYs7JWbGmShbi38gSxxXL, address:145.79.227.10:4419): protocol: rpc: dial tcp 145.79.227.10:4419: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401091332 (566h59m39.753876461s) [noise=true , address=145.79.227.12:4782 , piece_id=Q2WT5KUQ4UWHYNV5SW6XOJ4AMNPDLGIFWR2BVNAQGDJJ3ECNHAJQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ, address:145.79.227.12:4782): protocol: rpc: dial tcp 145.79.227.12:4782: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401101101 (566h59m39.753886229s) [noise=true , address=145.79.227.15:4890 , piece_id=SXSKTDJ6WKYGYLIHY7JPJU4CCIB57Y73COOERWFTPOVJDBAMHVAQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1gakF2Cctzc6EhFScfLfBHJdcDHqMBcey4mNoToEwbtk8Pwix5, address:145.79.227.15:4890): protocol: rpc: dial tcp 145.79.227.15:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401148540 (566h59m39.753933678s) [noise=true , address=145.79.227.14:4866 , piece_id=NRUKXI3FQCGQI5WSURFOD5MWY5L5OJUV3B7RZLX33CAN5TXBCK4Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:145.79.227.14:4866): protocol: rpc: dial tcp 145.79.227.14:4866: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401183265 (566h59m39.753968403s) [noise=true , address=145.79.227.13:4815 , piece_id=FKNQGYM6WHIOTFUJOGQPTVBUCCMKIJKM5634ORIXIRIKXGDLAX7Q, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401203854 (566h59m39.753988992s) [noise=true , address=145.79.227.2:4118 , piece_id=UMZVCTVPRDDERUTNRCRLPVCHJL6MSGHIGLEIXO66FQC7NP7RNCUQ, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:145.79.227.2:4118): protocol: rpc: dial tcp 145.79.227.2:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401243117 (566h59m39.754028245s) [noise=true , address=145.79.227.8:4355 , piece_id=3VMN7GJIJ4DSSRBC3EXPJLXGAO2KIFJ3HUZXPBN4GC7CXOYNSXRA, seg_pos={0 0}, upload=498993]: Done uploading piece. err: ecclient: upload failed (node:12mRuEpX7CMxWXVdHrTi3zF3v9t3LcqE6mJKzocajEvYHM52a7T, address:145.79.227.8:4355): protocol: rpc: dial tcp 145.79.227.8:4355: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:08:04.401288963 (566h59m39.754074102s) [seg_pos={0 0}, upload=498993]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Sep 22 05:08:04.415054225 (566h59m39.767839353s) [upload=498993]: Done uploading segments... Sep 22 05:08:04.415080705 (566h59m39.767865833s) [upload=498993]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 341.206792ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)